Method including generating and displaying a focus assist image indicating a degree of focus for a plurality of blocks obtained by dividing a frame of image signal

ABSTRACT

The present disclosure is directed to a computer system designed to (i) receive a series of images as input; (ii) compute a number of metrics derived from focus features and color separation features within the images; and (iii) evaluate the metrics to return (a) an identification of the most suitable z-layer in a z-stack, given a series of z-layer images in a z-stack; and/or (b) an identification of those image tiles that are more suitable for cellular based scoring by a medical professional, given a series of image tiles from an area of interest of a whole slide scan.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation of International PatentApplication No. PCT/EP2016/051903 filed Jan. 29, 2016, which claimspriority to and the benefit of U.S. Provisional Application No.62/110,215 filed Jan. 30, 2015. Each of the above patent applicationsare hereby incorporated by reference herein in their entireties.

BACKGROUND OF THE DISCLOSURE

Molecular pathology is the examination at a molecular level of the DNA,mRNA, and proteins that cause or are otherwise associated with disease.Gene amplification and/or overexpression have been identified as anindicator of patient prognosis in a variety of tumors or for determiningthose patients that should be provided certain treatments. For example,a certain type of breast cancer is associated with an over-abundance(e.g., over expression) of the human epidermal growth factor 2 (“HER2”)versus the number of chromosome 17s found in the cell. Sadly, thisalteration is also an independent prognostic factor predictive of poorclinical outcome and a high risk of recurrence. By detecting the numberof HER2 genes versus the number of chromosome 17s in a tissue sample,this particular type of breast cancer can be more readily identified andtreatment options can be evaluated.

In-situ hybridization can be used to look for the presence of a geneticabnormality or condition such as amplification of cancer causing genesspecifically in cells that, when viewed under a microscope,morphologically appear to be malignant. In situ hybridization (ISH)employs labeled DNA or RNA probe molecules that are anti-sense to atarget gene sequence or transcript to detect or localize targetednucleic acid target genes within a cell or tissue sample. ISH isperformed by exposing a cell or tissue sample immobilized on a glassslide to a labeled nucleic acid probe which is capable of specificallyhybridizing to a given target gene in the cell or tissue sample. Severaltarget genes can be simultaneously analyzed by exposing a cell or tissuesample to a plurality of nucleic acid probes that have been labeled witha plurality of different nucleic acid tags. By utilizing labels havingdifferent emission wavelengths, simultaneous multicolored analysis maybe performed in a single step on a single target cell or tissue sample.For example, INFORM HER2 Dual ISH DNA Probe Cocktail Assay from VentanaMedical Systems, Inc., is intended to determine HER2 gene status byenumeration of the ratio of the HER2 gene to Chromosome 17. The HER2 andChromosome 17 probes are detected using a two color chromogenic ISH informalin-fixed, paraffin-embedded human breast cancer tissue specimens.

Digital microscopy systems have been introduced wherein tissue samplesare prepared in a traditional manner, i.e. mounted on glass slides, butinstead of having the pathologist view the samples using a manuallycontrolled optical microscope, the slides are processed using digitalimaging equipment. In recent years, digital pathology has transformedfrom the use of camera-equipped microscopes to high-throughput digitalscanning of whole tissue samples. This development not only enablesvirtual storing and sharing of biological data, but it also improves theturnaround times for the pathologist and the patient.

The dramatic increase of computer power over the past decades, togetherwith the development of advanced image analysis algorithms, has allowedthe development of computer-assisted approaches capable of analyzing thebio-medical data and assisting in the diagnosis. Interpreting tissueslides manually is labor intensive, costly and involves the risk ofhuman errors and inconsistency, while using automated image analysis canprovide additional automatic, fast and reproducible analyses, assistingthe pathologist in making an accurate and timely diagnosis.

Challenges remain in computer-assisted diagnosis, including increasingthe accuracy and speed in providing a useful outcome, and at the sametime handling the enormous amount of data involved in digitalhistological samples. Thus, while advancements have been made, thereremains the need to further improve upon image acquisition and tobalance image quality with speed in image acquisition. Indeed, properlyfocused images having sharp, discernible features are needed for furtherdownstream processing, and failure to provide quality, focused imagescould lead to errors or ambiguous results. To date, however, onlygeneric algorithms have been developed to determine image focus. Forexample, previous approaches have derived sharpness metrics based solelyon a green channel of a color image. These generic approaches are not,however, able to consistently capture an image that provides both goodfocus and clear discrimination between differently colored features ofan image, both indispensable qualities for images undergoing cellularbased scoring. It would be desirable to have a new focus metric to helpselect the most suitable focus depth of a region of a tissue sample forfurther scanning and downstream processing. It is to be noted that forDual ISH, since the gene expression is manifested through dots, thescanning needs is performed at 40× resolution, while for most digitalpathology applications, a resolution of 20×0 is sufficient. Since ascanning resolution of 40× resolution is necessary for Dual ISH, animproved focus metric which ensures better quality at the 40× resolutionis required for Dual ISH.

In addition, automatic quality evaluation of a whole slide scan has beenanother challenging problem, and there are no known state-of-the-artmethods which can automatically detect “better quality” and“easy-to-score” regions, especially for Dual ISH scans. Thus, it wouldalso be desirable to have an automated and computationally efficient wayto assist a medical professional in assessing the quality of digitalimages of tissue samples, wherein the assessment accommodates differentfeatures appearing in the whole slide scanned image.

BRIEF SUMMARY OF THE DISCLOSURE

In one aspect of the present invention is a computer device configuredto analyze images from tissue samples comprising one or more processorsand at least one memory, the at least one memory storing non-transitorycomputer-readable instructions for execution by the one or moreprocessors to cause the one or more processors to execute instructionsto (i) compute a number of metrics derived from focus features and colorseparation features within the images of the tissue samples; and (ii)evaluate the metrics to return (a) an identification of the “mostsuitable” z-layer in a z-stack (as defined herein), given a series ofz-layer images in a z-stack; and/or (b) an identification of those imagetiles that are “more suitable” (as defined herein) for cellular basedscoring by a medical professional, given a series of image tiles from anarea of interest of a whole slide scan.

In some embodiments, the tissue sample is pretreated with two in situhybridization probes so as to provide samples, namely cells havingdifferently colored dots, e.g. black and red, signifying marked genes.In some embodiments, the metrics are derived by evaluating features ofblack dots and red dots present in the cells in each of the images. Insome embodiments, the metrics are computed by applying one or morefilters to the images received as inputs, where the filters adapt theimages such that focus and/or color separation features may be derived.Without wishing to be bound by any particular theory, it is believedthat any z-layer identified as the best layer comprises dots having goodfocus and clear discrimination between differently colored dots (e.g.black vs. red dots have high color separation). In addition, it isbelieved that image tiles identified as “more suitable” for scoring alsocomprise dots having good focus and clear discrimination betweendifferently colored dots (e.g. black vs. red dots).

In another aspect of the present disclosure is a computer system fordetermining the most suitable z-layer in a given z-stack, comprising oneor more processors and at least one memory, the at least one memorystoring non-transitory computer-readable instructions for execution bythe one or more processors to cause the one or more processors to:compute focus metrics and color separation metrics for each z-layerwithin a z-stack of images, each z-layer within the z-stackcorresponding to an image of a tissue sample; and evaluate the focusmetrics and color separation metrics to determine a most suitablez-layer within the z-stack. In some embodiments, the focus metricscomprise a focus quality score for each z-layer, and wherein the colorseparation metrics comprise a color separation quality score for eachz-layer. In some embodiments, the focus quality score for each z-layerand the color separation quality score for each z-layer areindependently computed within color spaces that are empiricallydetermined to provide the best focus and/or color separation for aparticular in situ hybridization assay.

In some embodiments, the evaluation of the focus metrics and colorseparation metrics comprises computing an absolute value metric anddetermining whether the absolute value metric is greater than, equal to,or less than a predetermined threshold value. In some embodiments, theabsolute value metric is an absolute value of the difference between thez-layer having best focus and the z-layer having best color separation.In some embodiments, the z-layer having best focus and the z-layerhaving best color separation are each independently computed by medianfiltering the focus quality scores and color separation quality scores,respectively, and then identifying a maximum value for the medianfiltered focus quality scores and a maximum value for the medianfiltered color separation quality scores. In some embodiments, if theabsolute metric is determined to be less than or equal to thepredetermined threshold value, instructions are provided to set the mostsuitable z-layer as a compromise layer metric, wherein the compromiselayer metric is an average value of the z-layer having best focus andthe z-layer having best color separation. In some embodiments, if theabsolute metric is determined to be greater than the predeterminedthreshold value, instructions are provided to evaluate whether the mostsuitable z-layer should be guided by focus features or color separationfeatures. In some embodiments, the evaluation of whether the mostsuitable z-layer should be guided by focus features or color separationfeatures is determined by comparing a layer focus comparator value to alayer color separation comparator value, whereby if the layer focuscomparator value is greater than the layer color separation value, themost suitable z-layer is set as the z-layer having best focus, andwhereby if the layer focus comparator value is less than the layer colorseparation value, the most suitable z-layer is set as the z-layer havingbest color separation.

In another aspect of the present disclosure is a computer-implementedmethod for determining the most suitable z-layer in a given z-stack,comprising computing focus metrics and color separation metrics for eachz-layer within a z-stack of images, each z-layer within the z-stack ofimages corresponding to an image of a tissue sample, and evaluating thefocus metrics and color separation metrics to determine a most suitablez-layer within the z-stack. In some embodiments, the focus metricscomprise a focus quality score for each z-layer, and wherein the colorseparation metrics comprise a color separation quality score for eachz-layer.

In some embodiments, the of the focus metrics and color separationmetrics comprises computing an absolute value metric and determiningwhether the absolute value metric is greater than, equal to, or lessthan a predetermined threshold value. In some embodiments, the theabsolute value metric is an absolute value of the difference between thez-layer having best focus and the z-layer having best color separation.In some embodiments, the z-layer having best focus and the z-layerhaving best color separation are each independently computed by medianfiltering the focus quality scores and color separation quality scores,respectively, and then identifying a maximum value for the medianfiltered focus quality scores and a maximum value for the medianfiltered color separation quality scores. In some embodiments, if theabsolute metric is determined to be less than or equal to thepredetermined threshold value, instructions are provided to set the mostsuitable z-layer as a compromise layer metric, wherein the compromiselayer metric is an average value of the z-layer having best focus andthe z-layer having best color separation. In some embodiments, if theabsolute metric is determined to be greater than the predeterminedthreshold value, instructions are provided to evaluate whether the mostsuitable z-layer should be guided by focus features or color separationfeatures. In some embodiments, the evaluation of whether the mostsuitable z-layer should be guided by focus features or color separationfeatures is determined by comparing a layer focus comparator value to alayer color separation comparator value, whereby if the layer focuscomparator value is greater than the layer color separation value, themost suitable z-layer is set as the z-layer having best focus, andwhereby if the layer focus comparator value is less than the layer colorseparation value, the most suitable z-layer is set as the z-layer havingbest color separation.

In another aspect of the present disclosure is a computer-implementedmethod for the automated evaluation of image tiles derived from a wholeslide scan comprising: (a) computing a plurality of focus features and aplurality of color separation features for each individual image tile;(b) deriving a focus quality score from the plurality of focus featuresand a color separation quality score from the plurality of colorseparation features; and (c) identifying digital image tiles moresuitable for downstream processing based on the focus quality score andthe color separation quality score. In some embodiments, the pluralityof focus features are selected from (i) a mean/median of the 0th DoGlayer for all dot pixels in the image tile; (ii) a mean/median of the0th DoG layer for all black dot pixels in the image tile; (iii) amean/median of the max DoG values of the 0th layer for all dot blobs inthe image tile; and (iv) a mean of the top DoG values for the 0th layerfor all dot blobs in the image tile.

In some embodiments, the plurality of color separation features areselected from (i) Amax, (ii) Asigmax; (iii) a maximum in an unmixedblack channel; (iv) a gradient value for at least one channel selectedfrom the group consisting of a luminance channel, an unmixed redchannel, an A channel, an Asig channel, and a green channel; (v) DoGvalues in at least one of a channel selected from the group consistingof a green channel, an A channel, and an optical density domainabsorbance channel; and (vi) color domain features based on modelingcolor information in terms of ellipses.

In some embodiments, the method further comprises generating a heat map.In some embodiments, the method further comprises generating an overlay,where the overlay indicates digital image tiles more suitable fordownstream processing. In some embodiments, the focus quality score andcolor separation quality score are computed only for those tiles havingat least one dot corresponding to a first in situ hybridization signaland at least one dot corresponding to a second in situ hybridizationsignal (e.g. a red signal and a black signal).

In another aspect of the present disclosure is a computer system for theautomated evaluation of image tiles derived from a whole slide scancomprising one or more processors and at least one memory, the at leastone memory storing non-transitory computer-readable instructions forexecution by the one or more processors to cause the one or moreprocessors to: (a) compute a plurality of focus features and a pluralityof color separation features for each individual image tile; (b) derivea focus quality score from the plurality of focus features and a colorseparation quality score from the plurality of color separationfeatures; and (c) identify digital image tiles more suitable fordownstream processing based on the focus quality score and the colorseparation quality score. In some embodiments, the plurality of focusfeatures are selected from (i) a mean/median of the 0th DoG layer forall dot pixels in the image tile; (ii) a mean/median of the 0th DoGlayer for all black dot pixels in the image tile; (iii) a mean/median ofthe max DoG values of the 0th layer for all dot blobs in the image tile;and (iv) a mean of the top DoG values for the 0th layer for all dotblobs in the image tile.

In some embodiments, the plurality of color separation features areselected from (i) Amax, (ii) Asigmax; (iii) a maximum in an unmixedblack channel; (iv) a gradient value for at least one channel selectedfrom the group consisting of a luminance channel, an unmixed redchannel, an A channel, an Asig channel, and a green channel; (v) DoGvalues in at least one of a channel selected from the group consistingof a green channel, an A channel, and an optical density domainabsorbance channel; and (vi) color domain features based on modelingcolor information in terms of ellipses.

In some embodiments, instructions are provided to compute a heat map. Insome embodiments, instructions are provided to generate an overlay,where the overlay indicates digital image tiles more suitable fordownstream processing. In some embodiments, the focus quality scores andcolor separation quality score are computed only for those tiles havingat least one dot corresponding to a first in situ hybridization signaland at least one dot corresponding to a second in situ hybridizationsignal.

In another aspect of the present disclosure is a computer deviceconfigured to analyze images from tissue samples comprising one or moreprocessors and at least one memory, the at least one memory storingnon-transitory computer-readable instructions for execution by the oneor more processors to cause the one or more processors to (i) receive aseries of images as input, each image corresponding to a z-layer in az-stack; (ii) run a focus assessment module to compute a number ofmetrics derived from focus features and color separation features withinthe images; and (iii) output a most suitable z-layer in a z-stack ofimages based on the computed metrics. In some embodiments, a qualityassessment module is run and an identification of tiles within the inputimage that are more suitable for downstream processing are provided asoutputs.

In another aspect of the present disclosure is a computer system fordetermining a z-layer that, compared to other z-layers in a givenz-stack, comprise features that are well focused and have clearlydiscernible color features such that differently colored features may berecognized from each other, comprising one or more processors and atleast one memory, the at least one memory storing non-transitorycomputer-readable instructions for execution by the one or moreprocessors to cause the one or more processors to execute instructionsto: retrieve a series of images from a z-stack of images, each imagecorresponding to a different z-layer in the z-stack, and wherein theimages are captured from a portion of a tissue specimen pre-treated withtwo in situ hybridization probes; compute focus metrics and colorseparation metrics for each z-layer in the z-stack, wherein the focusmetrics are derived from focus features of stained regions within theimages and wherein the color separation metrics are derived from colorseparation features of stained regions within the images; and determinethe z-layer that, compared to other z-layers in the given z-stackcomprises features that are well focused and have clearly discerniblecolor features, based on evaluation of the focus metrics and colorseparation metrics.

In another aspect of the present disclosure is a computer device orsystem for determining the most suitable z-layer in a given z-stackcomprising one or more processors and at least one memory, the at leastone memory storing non-transitory computer-readable instructions forexecution by the one or more processors to cause the one or moreprocessors to execute instructions to: retrieve a series of images froma z-stack of images, each image corresponding to a different z-layer inthe z-stack, wherein the images are derived from a portion of a tissuespecimen pre-treated with two in situ hybridization probes; computefocus metrics and color separation metrics for each z-layer in thez-stack, wherein the focus metrics are derived from focus featureswithin the images and wherein the color separation metrics are derivedfrom color separation features within the images; determine a mostsuitable z-layer within the z-stack based on the evaluation of the focusmetrics and color separation metrics; and output the most suitablez-layer for further use in further downstream processes.

In some embodiments, the focus metrics comprise a focus quality scorefor each z-layer in the given z-stack and the color separation metricscomprise a color separation quality score for each z-layer in the samez-stack. In some embodiments, the focus metrics further comprise anidentification of a z-layer having the best focus as compared with otherz-layers in the z-stack; and the color separation metrics furthercomprise an identification of a z-layer having the best color separationas compared with other z-layers in the z-stack. In some embodiments, thesystem further comprises instructions to compute the z-layer having bestfocus and the z-layer having best color separation by applying analgorithm to independently median filter the focus and color separationquality scores, respectively, and then identify a maximum value for themedian filtered focus quality scores and separately identify a maximumvalue for the median filtered color separation quality scores. In someembodiments, the system further comprises instructions to choose acompromise layer, wherein the compromise layer chosen is an averagevalue of the z-layer having best focus and the z-layer having best colorseparation; and instructions to compute an absolute value metric,wherein the absolute value metric is an absolute value of the differencebetween a value of the z-layer having best focus and a value of thez-layer having best color separation.

In another aspect of the present disclosure is a computer system fordetermining the most suitable z-layer in a given z-stack comprising amemory for storing a sequence of program instructions and/or images; aprocessor communicatively coupled with the memory that is configured toexecute the program instructions; wherein the program instructionsretrieve input images corresponding to a series of z-layers in a givenz-stack of images; compute a focus score and a color separation scorefor each layer in a z-stack; determine a z-layer having best focus basedon the focus scores of all of the z-layers in the z-stack; determine az-layer having best color separation based on the color separationscores of all of the z-layers in the z-stack; determine whether theabsolute value of a difference between the z-layer having best focus andthe z-layer having best color separation is greater than, equal to, orless than a pre-defined threshold; where if the absolute difference isgreater than the threshold, the most suitable z-layer for furtherprocessing is computed based on whether focus features or colorseparation features provide a better indication of image scoreability(i.e. whether the best focused layer is easiest to score, or the layerwith the best color separation is easiest to score); and if the absolutevalue difference is less than or equal to the threshold, compute thebest z-layer based on a combination of focus features and colorseparation features; and outputting the most suitable z-layer forfurther processing. In some embodiments, the focus scores and colorseparation scores are each computed based on an empirically determinedcolor space based on features that best describe the focus andseparately features that best describe color separation of the regionsof interest in the images. In some embodiments, the focus metrics andcolor separation metrics are derived by evaluating features of blackdots and red dots in the cells present in the images, wherein the imagesare derived from tissue samples pretreated with two different in situhybridization probes (e.g. for Dual ISH). In some embodiments, the blackdots and red dots are present in the images of cell nuclei whoseoriginating tissue was treated with a dual color, dual hapten assay,such that the red dots represent detection of chromosome 17 while theblack dots represent detection of the HER2 gene.

In another aspect of the present disclosure is a quality assessmentmodule for a computer-based system for determining particular tiles of awhole slide scan which may be better suited for scoring by a pathologistcomprising one or more processors and at least one memory, the at leastone memory storing non-transitory computer-readable instructions forexecution by the one or more processors to cause the one or moreprocessors to execute instructions to (a) analyze the image tiles inorder to compute a plurality of focus features and a plurality of colorseparation features; (b) determine a focus quality score for each imagetile based on the focus features and a color separation quality scorefor each image tile based on the color separation features; (c) outputan identification of those image tiles that may be more suitable forcellular based scoring by a medical professional. In some embodiments,instructions are provided to filter the respective quality scores toreturn a user specified percentage of image tiles having the highestquality for scoring by a pathologist. In some embodiments, the systemprovides instructions for the generation of an image overlay which issuperimposed over at least a portion of the whole slide scan image andwhich visually reflects the image tiles that may be more suitable forscoring by a pathologist. The quality assessment module described hereinmay be run by a computer system contemporaneously while the system scansand/or captures images, including while computing metrics and performingevaluations to determine the most suitable z-layer. In some embodiments,the focus quality scores and color separation quality scores of thequality assessment module may be used to compute a most suitable z-layerin a z-stack, although on current computer systems, computation andevaluation in this manner is slower than the method described herein forz-layer identification. In some embodiments, the quality assessmentmodule may be run on images scanned using the focus metric moduledescribed herein. In other embodiments, the quality assessment modulemay be run on images scanned using a prior art focus module.

In a further aspect, the disclosure relates to an image analysis methodfor determining one of a plurality of z-layers in a z-stack of digitalimages. The image of the determined z-layer is to be used for computingtissue-related or cell-related scores. The z-stack of digital imagesdepicts different layers of a single tissue sample located on a slide.The z-stack images have been acquired by varying the focus of an opticalsensor, e.g. by varying the focus of the optical sensor in steps andcapturing a new image for each step. The method comprises:

for each of the z-layers, computing focus metrics, each focus metricbeing an indicator of focus quality of the z-layer and being derivedfrom image data of said z-layer;

for each of the z-layers, computing color separation metrics, each colorseparation metric being an indicator of color separation quality of thez-layer and being derived from image data of said z-layer; and

evaluating the focus metrics and the color separation metrics anddetermining the one of the z-layers to be used for computingtissue-related or cell-related scores in dependence on the focus metricsand color separation metrics for each of the z-layers.

Applicants have discovered a superior method of determining the mostsuitable z-layer in a given z-stack of images. Indeed, when compared toprior art methods, the present method for z-layer identification isconsistently superior and experimentation has shown that the new methodis able to accurately determine the most suitable z-layer in more casesas compared with the prior art methods. Moreover, the focus metricdisclosed herein is able to return z-layers that consistently meet therequirements of good focus and high discriminability between the twodifferent dot colors (red and black). Thus, the method disclosed hereinprovides for a fast focus metric that returns the most suitable z-layerin a significantly higher number of cases as compared to the prior artfocusing methods.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided to the Office upon request and thepayment of the necessary fee.

FIG. 1 shows a computer-based system for analyzing specimens inaccordance with an embodiment of the disclosed technology;

FIG. 2 provides a flowchart showing the steps of determining a mostsuitable z-layer in a z-stack of images (focus assessment module);

FIG. 3 provides a flowchart showing the steps of determining suitabletiles for further processing (quality assessment module);

FIG. 4 shows a tissue specimen captured in a whole slide scan and alsoshows the results of application of an algorithm to determine an area ofinterest of the whole slide scan (note that here the slide thumbnail isof an H&E image and not of a Dual ISH slide, and this thumbnail isprovided for illustrative purpose only to give a visual feel of thetissue region extraction problem);

FIG. 5A shows an example of a heat map, where the shading of each tileindicates the quality (ease of scoring) of the tiles;

FIG. 5B is an exploded view of an area within FIG. 5A, again showingdifferently shaded tiles;

FIG. 6 provides an example of a dual ISH assay to detect HER2 havingSISH signal dots and Red ISH signal dots;

FIG. 7A shows an example of ellipse fitting based on features indicatingred versus black separation;

FIG. 7B provides a visual representation of red versus black colorseparation based on the ellipse fitting of FIG. 7A where the black andred dots are labeled; and

FIG. 8 provides an example of how a change of z-layer can cause betterred versus black separation, which can be measured by the ellipsefitting method.

FIG. 9 depicts a plot that illustrates the predictive power for anexample focus metric having a high predictive power in respect toidentifying images having high focus quality.

FIG. 10a depicts the representation of 6 dots of a first color and 6dots of a second color in an optical density domain coordinate system.

FIGS. 10b and 10c depict the projections for finding the azimuth andelevation angles for the vectors of FIG. 10 a.

FIG. 11 depicts the representation of the 8 vectors of FIGS. 10b and 10cin an azimuth-elevation plot.

DETAILED DESCRIPTION

As used herein, the singular terms “a,” “an,” and “the” include pluralreferents unless the context clearly indicates otherwise. Similarly, theword “or” is intended to include “and” unless the context clearlyindicates otherwise.

The terms “comprising,” “including,” “having,” and the like are usedinterchangeably and have the same meaning. Similarly, “comprises,”“includes,” “has,” and the like are used interchangeably and have thesame meaning. Specifically, each of the terms is defined consistent withthe common United States patent law definition of “comprising” and istherefore interpreted to be an open term meaning “at least thefollowing,” and is also interpreted not to exclude additional features,limitations, aspects, etc. Thus, for example, “a device havingcomponents a, b, and c” means that the device includes at leastcomponents a, b and c. Similarly, the phrase: “a method involving stepsa, b, and c” means that the method includes at least steps a, b, and c.Moreover, while the steps and processes may be outlined herein in aparticular order, the skilled artisan will recognize that the orderingsteps and processes may vary.

A “tissue sample” as used herein is any biological sample that isobtained from a human or animal body for anatomic pathology. Forexample, a tissue sample may be derived from breast tissue, lung tissue,prostate tissue, etc. and may comprise samples derived from tumors,suspected tumors, or from healthy tissue. Other examples of tissuesamples and specimens are their preparation are disclosed herein. Thetissue sample may be treated in an assay with one or more stains toassist in the identification of structures (e.g. vessels, cells, etc.)within the sample.

An “unmixed image” as used herein encompasses a grey-value or scalarimage obtained for one channel of a multi-channel image. By unmixing amulti-channel image one unmixed image per channel is obtained.

The present disclosure describes a focus algorithm (“focus metricmodule”) which Applicants believe to be better capable of capturing themost suitable z-layer in a given z-stack of images. The presentdisclosure sets forth the goal of being to obtain image scans of tissuesamples where image features are in good focus and where features ofdifferent colors (e.g. corresponding to first and second in situhybridization signals) may be clearly discernible from each other, suchthat the red and black dots may be visually and algorithmicallydistinguishable.

The present disclosure also provides a “quality assessment” module whichcomputes different metrics based on focus quality and allows fordiscrimination between differently colored image objects for aparticular image tile and, given this information, provides guidance toa medical professional, e.g. in the form of a heat map, showing thosetiles which are believed to be more suitable for further processing oranalysis. Indeed, the quality assessment module provides anidentification of recommended tiles regarded by the algorithm as havinga “better quality” and the medical professional may concentrate on cellselection and scoring on those tiles/regions. The quality assessmentmodule is intended to help and guide the pathologist or other medicalprofessional in selecting easy-to-score regions in a whole slide scan(e.g. as required by dual ISH protocols).

The quality assessment module and separate focus metric module areindependent modules. In some embodiments, the focus metric module runsinside the scanner during the real-time process of scanning while the“quality assessment” module runs on all the image tiles in an “offline”mode once the whole slide scan has been generated.

At least some embodiments of the technology disclosed herein relate tocomputer systems and methods for analyzing digital images from tissuesamples pretreated with in situ hybridization probes. The samples may bea breast tissue sample processed according to an in situ hybridization(“ISH”) protocol, as known in the art. In other embodiments, thespecimen is a gastric tissue sample including, but not limited tostomach tissue, esophageal tissue, gastro-esophageal junction tissue,intestinal tissue, and colon tissue, again processed according to an ISHprotocol. While specific examples herein may refer to breast tissue,these and other tissues are contemplated.

The ISH protocol provides visualization of specific nucleic acidsequences (e.g., DNA, mRNA, etc.) in frozen tissue sections,fixed/paraffin embedded tissue sections, or other cell preparations byhybridizing complementary strands of nucleotides (e.g., probes) to thesequence of interest. The ISH protocol can include, without limitation,a dual SISH and Red ISH protocol, single Red ISH protocol, single SISHprotocol, or the like.

Although exemplary embodiments described herein disclose the applicationof a dual ISH probe to breast tissue, ultimately for the detection ofthe expression of the HER2 gene in the cells contained therein, it willbe appreciated that the technology can be used to analyze images ofother tissue samples treated with other probes and/or assays to detectother genes or portions thereof in cells, as well as other features ofinterest. Indeed, certain embodiments disclose application of dual ISHscans, taken at 40× resolution, where over- or under-expression of agene depends on the identification of stained dots, where black dots andred dots are expressed through HER2 and Chr-17 markers, respectively.While the present disclosure pertains to identifying the most suitablelayer in a z-stack and/or providing a quality analysis of image tiles,images captured of tissue samples are ultimately evaluated in furtherdownstream processes, e.g. to determine a ratio of black dots to reddots for HER2 detection, as described further in Example 1 herein.

Although tissue samples on microscope slides appear flat(two-dimensional), they are three-dimensional objects having aconsiderable amount of variation in thickness. This is apparent when theslides are considered at microscopic levels where different imagescorresponding to different planes (z-axis) may be captured to create a(volume stack) z-stack of images. For example, for a 25-micron thicksample, it is possible to adjust the distance from a sample to anobjective lens (z-axis) in about five micron increments to visualizeand/or digitally capture five different images, with each image being ina different plane and having a different focus depth. Of course, thesample may be of any thickness and any incremental focus depth may bechosen (e.g. 1 micron, 0.5 micron, 0.25 micron, 2 micron spacings). Itis believed that for the specific problem of better Dual ISH scanning, aslight difference in focus depth of about 0.25 microns (which is, e.g.,the finest resolution for the Coreo and HT scanners for a z-scan) maylead to noticeable changes in image sharpness and focus, and hence thedifference between consecutive z-layers for Dual ISH scanning isgenerally kept at the finest possible z-resolution of about 0.25 micron.In some embodiments, the number of planes digitally captured in anygiven z-stack ranges from about 10 to about 20. In some embodiments, thenumber of planes digitally captured is about 15.

As used herein, a set of images acquired from one location on thespecimen at different focus depths is referred to as a “z-stack” andeach individual image in a given z-stack is referred to as a “z-layer.”Thus, each z-layer in a given z-stack represents an image having adifferent focal plane within a tissue sample location.

As used herein, the term “tile” refers to a region of a whole slide scanor an area of interest having an (x,y) pixel dimension (e.g. about 300×about 300 pixels). Tile size/area selection is described further herein.

By way of example, a computer-based specimen analyzer for analyzingspecimens is shown in FIG. 1. The skilled artisan will appreciate thatother computer systems may be utilized and that the computer systemsdescribed herein may be communicatively coupled to additionalcomponents, e.g. analyzers, scanners, etc. Some of these additionalcomponents and the various computers that may be utilized are describedfurther herein.

In general, the imaging apparatus 12 can include, without limitation,one or more image capture devices. Image capture devices can include,without limitation, a camera (e.g., an analog camera, a digital camera,etc.), optics (e.g., one or more lenses, sensor focus lens groups,microscope objectives, etc.), imaging sensors (e.g., a charge-coupleddevice (CCD), a complimentary metal-oxide semiconductor (CMOS) imagesensor, or the like), photographic film, or the like. In digitalembodiments, the image capture device can include a plurality of lensesthat cooperate to prove on-the-fly focusing. A CCD sensor can capture adigital image of the specimen. One method of producing a digital imageincludes determining a scan area comprising a region of the microscopeslide that includes at least a portion of the specimen. The scan areamay be divided into a plurality of “snapshots.” An image can be producedby combining the individual “snapshots.” In some embodiments, theimaging apparatus 12 produces a high-resolution image of the entirespecimen.

The computer system 14 can include a desktop computer, a laptopcomputer, a tablet, or the like and can include digital electroniccircuitry, firmware, hardware, memory, a computer storage medium, acomputer program, a processor (including a programmed processor), or thelike. The illustrated computing system 14 of FIG. 1 is a desktopcomputer with a screen 16 and a tower 18. The tower 18 can store digitalimages in binary form. The images can also be divided into a matrix ofpixels. The pixels can include a digital value of one or more bits,defined by the bit depth. The network 20 or a direct connectioninterconnects the imaging apparatus 12 and the computer system 14.

Focus Assessment Module

In a first aspect is an improvement to the overall scanning workflow forDual ISH by computing focus scores which ensures that for Dual ISHz-stacks, the best layer (as defined further below) is more likely to bepicked as compared with prior art methods. In this disclosure, weexplain how focus score is computed (since scanning is a real timeprocess, the focus metric computation needs to be relatively fast), whatthe intuition is behind the new focus score computation, and whatexperiments have been conducted to ensure that the new focus metric isbetter than old focus metrics.

Therefore, in one aspect of the present disclosure is a computer-baseddevice or system for determining the most suitable z-layer in a z-stackfor further processing, e.g. obtaining a high resolution scan of justthe single z-layer in the given z-stack having features that are wellfocused and where differently colored features may be cleareddiscriminated from each other (i.e. good color separation, e.g. toclearly discriminate first ISH signals from second ISH signals or reddots from black dots). The steps for determining the most suitablez-layer are described herein.

As used herein, the term “color separation” means clear and discernibledifferentiation between the colors of features in an image. Colorseparation is represented graphically in FIG. 8 where an image havingbetter color separation (top image) is compared to one having inferiorcolor separation (bottom image), such that when the black dots and reddots of the better color separated image are graphed in the opticaldensity domain (FIG. 8) the RGB colors are converted to the opticaldensity domain, and from there mapped to spherical coordinates, and herethe azimuth and elevation angles, derived from spherical coordinaterepresentation, are plotted as the red-vs-black color separation can bebetter explained using these two axes), the red and black dots areseparated from each other and have minimal overlap. As that overlapincreases, depicted in the lower graphic representation of FIG. 8, thediscernibility between red and black dots decreases.

As used herein, the term “most suitable z-layer” is a z-layer that,compared to other z-layers in a given z-stack, contains features thatare well focused and have clearly discernible color features such thatdifferently colored features may be recognized from each other. Forexample, if an image comprises black dots and red dots (e.g. frompretreatment of the tissue sample with dual ISH probes), the mostsuitable layer will have dots that are well focused, and the black andred dots will be clearly identifiable and discernible from each other,optimally such that the black dots appear black and the red dots appearred. Of course, this concept may be applied to any assay signals, notjust red and black signals.

Prior to determining the most suitable z-layer, in some embodiments, anarea of interest (AOI) of a tissue specimen of a whole slide scan isfirst found by running an AOI module on the computer system, as known tothose of skill in the art. An example of a tissue specimen where an AOIis computed is shown in FIG. 4. The AOI detection module, in someembodiments, returns a probability map, where each pixel denotes howlikely it is to belong to tissue as compared to glass. Focus points arethen allocated based on where the probability of tissue being detectedis higher. Then tiles are considered around each focus point. For eachtile, z-stacking is done with the aim of finding the most suitablez-layer in a given z-stack. After the most suitable z-layer is found(e.g. by using the focus metric module described herein) for each tilearound a focus point, 2D interpolation is performed to estimate the bestz-layer for intermediate tiles. Given this, a full scanned image usingthe interpolated z-layers may be captured.

In some embodiments, program instructions are run to retrieve and/orinput a series of images, with each image corresponding to a particularz-layer in a given z-stack of images. Program instructions are then runto determine a number of metrics for each image; the metrics are basedon identifiable features, such as focus features and color separationfeatures. Once the focus and color separation metrics are determined,the instructions are then executed to evaluate the metrics and, ingeneral, determine whether focus metrics, color separation metrics, or acombination of focus and color separation metrics better guide theselection of the most suitable z-layer within a given z-stack.

More specifically, the computer systems 14 include one or moreprocessors that are programmed with a series of computer-executableinstructions that are stored in a memory. With reference to theflowchart of FIG. 2, when executed the instructions cause the one ormore processors and/or the memory of the computer system to receive aseries of digital images (step 210), wherein each digital imagecorresponds to a particular z-layer in a given z-stack of scannedimages. The computer system then executes instructions that cause one ormore of the processors to compute focus metrics and color separationmetrics for each z-layer in the z-stack, wherein the metrics are derivedfrom focus features and color separation features, respectively withinthe images. In some embodiments, the metrics include focus qualityscores and color separation quality scores for each z-layer in the givenz-stack (step 220). In some embodiments, the color separation scores arered-versus-black (R/B) separation quality scores. Additional metricscomputed include a z-layer having the best focus as compared with otherz-layers in the given z-stack 230; a z-layer having the best colorseparation as compared with other z-layers in the same z-stack (step230); a compromise layer metric (step 240); and an absolute value metric(step 240). Each of these metrics, including how they are derived andevaluated to determine the most suitable z-layer for further processingare described herein.

In some embodiments, each of the focus and color separation metrics arecomputed in an optimized color space, e.g. the color space that isbelieved to provide the best quality focus score or color spaceseparation quality score. In some embodiments, the focus and colorseparation metrics are each computed in a color space optimized forstains or assays in which the tissue specimens were subjected. In someembodiments, the derived images are obtained based on a function appliedon the three color channels (red, green, and blue) where the bestfunction used to combine the color channels has been empiricallydetermined (as applied to dual ISH and as could be applied to other ISHprotocols utilizing different chromogens, etc.). This is in contrast tothe prior art where only the green channel was utilized.

The most common representation of color in digital images is the RGBmodel, referring to the color channels red, green, and blue. For the RGBmodel, the color of any particular pixel can be represented by athree-dimensional vector (r, g, b) that provides the respective colorintensities. There are different images derived from the RGB color spacewhich are linear combinations (or other functions) of the RGB colorspace and, it is believed, that a particular, optimized linearcombination provides improved focus quality scores and color separationquality scores as compared with the prior art (where quality scores werecomputed using sum of squared difference based cost computed on Greenchannel, which was not optimized for Dual ISH and which could notcapture the double constraints of good focus and good red versus blackcolor separation). In some embodiments, the optimization has beenempirically conducted by studying different possible linear combinationsof the form: Red+a*Green+b*Green, where a and b are both varied from −1to 1 in steps of 0.2, and the 3 color channels have been considered intheir original 8-bit and in optical density domain representation.

In some embodiments, the color channels for computation of the focus andcolor separation metrics are empirically determined. For example, byusing ground truth data collected by manually assigning focus and colorseparation scores to z-stacks, the best color channels for estimatingfocus and color separation were empirically determined. The “groundtruthing” applied is described in Examples 2 and 3 herein. As describedin Example 2, empirical determination of the color space which providedoptimized quality scores were computed with a 2-pixel difference basedmetric on (R+a*G+b*B), where −1<=a<=1, and −1<=b<=1, and the experimentswere conducted in both the RGB domain and in the optical density domain(optical density representation (ROD, GOD, BOD) of a pixel (R,G,B) in0-255 (8 bit) range is: ROD=−log(R/BK); GOD=−log(G/BK); BOD=−log(B/BK);where BK was an assumed background intensity of 240.

In the context of dual ISH for the detection of HER2 where black and reddots are identifiable features, it is believed that the best imagechannel to use for the focus quality score computation is(R−0.75*G−1*B), where all the channels are computed in the opticaldensity domain. For the same dual ISH protocol, it is believed that thebest image channel to use for the color separation score computation(here a red-vs-black separation score computation), is (R−0.5*G−0.5*B),where all channels are in RGB (each pixel has values in [0-255]) domain.The skilled artisan will be able to empirically determine other suitablecombinations of image channels for other assays, including dual ISHassays utilizing detectable markers having any differing and discerniblewavelengths. Of course, the skilled artisan will also be able to developalgorithms to determine optimized derived images, whereby the algorithmsmay be run prior to focus quality score computation and color separationquality score computation without changing the effect of the presentlydisclosed disclosure. It is believed that the optimization basicallyobtains a “better” function to combine the information from red, greenand blue channels to a single derived image; where a “better” functionis one where the cost, such as sharpness of focus in our example,computed on the derived image obtained using that function, is higherthan the same cost computed on another derived image obtained using anyother function.

To determine the focus score (step 220), a sum of squares differencebased cost function is utilized, whereJ=ΣxΣy[(|R(x,y)−R(x,y−2)|^2)−0.75(|G(x,y)−G(x,y−2)|^2)−1(|B(x,y)−B(x,y−2)|^2)];for all x,y, where R(x,y) refers to the pixel in the x-th row andy-column of the red optical density channel; G(x,y) refers to the pixelin the x-th row and y-column of the green optical density channel; andB(x,y) refers to the pixel in the x-th row and y-column of the blueoptical density channel, where red, green and blue channels have beenderived from the source RGB image, and these 3 color channels are thenconverted to their optical density representation. For the old focusmethod, the focus score is computed as: J=ΣxΣy(|G(x,y)−G(x,y−2)|^2);thus the sum of squared difference between every two consecutive pixelsis used, considering only the green channel.

Likewise, to determine the color separation score 220, a sum of squaresdifference based cost function is utilized, whereJ=ΣxΣy[(|R(x,y)−R(x,y−2)|^2)−0.5(|G(x,y)−G(x,y−2)|^2)−0.5(|B(x,y)−B(x,y−2)|^2)];

for all x,y, where R(x,y) refers to the pixel in the x-th row andy-column of the red channel; G(x,y) refers to the pixel in the x-th rowand y-column of the green channel; and B(x,y) refers to the pixel in thex-th row and y-column of the blue channel, where red, green and bluechannels have been derived from the source RGB image.

Once the focus quality score values and color separation quality scorevalues (step 220) are determined for each z-layer in the given z-stack,the computer system then executes instructions that cause one or more ofthe processors to evaluate the metrics and determine whether focusmetrics, color separation metrics, or a combination of focus and colorseparation metrics should guide selection of the most suitable z-layerin a given z-stack (e.g. steps 260 through 290).

In some embodiments, further metrics for evaluation include (1) thez-layer having the best focus as compared with other z-layers in thez-stack; (2) the z-layer having the best color separation as comparedwith other z-layers in the same z-stack; (3) a compromise layer metric;and (4) an absolute value metric.

In some embodiments, the computer system further comprises instructions(step 230) to compute the z-layer having best focus and the z-layerhaving best color separation by applying an algorithm to independentlymedian filter the focus and color separation quality scores and thenidentify a maximum value for the median filtered focus quality scoresand a maximum value for the median filtered color separation qualityscores.

For example, the z-layer having the best focus may be achieved medianfiltering the focus vectors (the focus quality score of each z-layer inthe z-stack) using, e.g. a 3×1 window, and then determining the maximumto obtain the z-layer having best focus (LF). Those of skill in the artwill be able to adjust a window size as needed. Likewise, the z-layerhaving the best color separation may be achieved by the computer medianfiltering the color separation vectors (the color separation qualityscore of each z-layer in the z-stack) using, e.g. a 3×1 window, and thendetermining the maximum to obtain the z-layer having best colorseparation.

Program instructions (step 240) are also executed to calculate acompromise layer metric (L) which is determined by averaging the z-layerhaving best focus value (LF) and the z-layer having best colorseparation value (LBR), e.g. L=(LF+LRB)/2.

Program instructions (step 240) are then further executed to compute anabsolute value metric, wherein the absolute value metric is an absolutevalue of the difference between the z-layer having best focus and thez-layer having best color separation, e.g. absolute valuemetric=abs|LF−LRB|. Given the various metrics, the computer system mayevaluate whether the identification of a most suitable z-layer should beguided by focus features, color separation features, or a combinationthereof.

In some embodiments, the computer system receives instructions tocompare the absolute value metric to a pre-defined threshold (step 250).For example, the computer system may receive instructions to evaluatewhether the absolute value metric is greater than, equal to, or lessthan a pre-defined threshold, i.e. abs|LF−LRB|>threshold;abs|LF−LRB|<threshold; or abs|LF−LRB|=threshold.

The threshold may be a maximum allowed difference between a best layerfor focus and a best layer for color separation. In some embodiments,the threshold is set to a integer less than half of the total number ofz-layers. For example, given a z-stack comprising between about 12 toabout 16 z-layers, the threshold may be between about 6 to about 8. Inother embodiments, the threshold ranges from about 4 to about 8. In oneembodiment, the threshold is 6.

If the computer determines that the absolute value metric is less thanor equal to the threshold (step 270), then the computer sets the mostsuitable z-layer in a given z-stack as the value of the compromise layermetric (L) (step 290). The idea is that a combination of focus metricsand color separation metrics guides the determination of the mostsuitable layer since the spacing in the z-axis between z-layers in thisinstance is small (in scanning experiments, the z-layer differencebetween successive layers is kept at about 0.25 microns).

On the other hand, if the absolute value metric is greater than thethreshold (step 260), instructions are then executed (step 280) todetermine whether a focus feature or a color separation feature is amore important feature in identifying the most suitable z-layer. Forexample, a layer focus comparator (LFC) may be compared to a referencefocus value (e.g. LFC=(focus at LF)/focus reference)) and a layer colorseparation comparator (LRBC) may be compared to a color separationreference value (e.g. LRBC=(Red-black-separation atLRB)/Red_black_reference). When compared, if the LFC value is greaterthan the LRBC value, then the computer sets the z-layer having bestfocus (LF) as the most suitable z-layer; otherwise, the computer setsthe most suitable z-layer as the z-layer having best color separation(LRB) (step 285).

The focus_reference and red-black-separation reference values arepredetermined values and, in general, are average focus scores per tileand average color separation scores per tile, respectively. In someembodiments, the focus_reference is 0.175 and the Red_black_reference is7.5, and these values were empirically derived through experiments,after collecting focus and red-vs-black separation scores from many goodquality tiles and poor quality tiles.

If any of LF, LRB, or L are not integers, the computer is instructed topick the next nearest integer (next nearest z-layer) whichever producesa better score with respect to the metric at hand (better focus, bettercolor separation, or both). When focus and color separation values peakfor different layers, preference is given to the next nearest z-layerhaving a better color separation. For example, if L is 2.3 and the nextnearest z-layer representing best color separation is 3, not 2, then thecomputer is instructed to choose 3 over 2. The intuitive explanation forthis is that the final task, for a trained pathologist, is to detect andcount the red and black dots (dots being of good focus only indicatethat they can easily be detected). However, when the color separation isbetter, it is easier to distinguish red and black dots and hence, scorethe Dual ISH image.

After the most suitable z-layer value is returned, instructions areprovided, in some embodiments, to scan a tissue sample at the particularregion identified and at a focus depth corresponding to the mostsuitable z-layer value. The image captured, e.g. a high resolutionimage, may be used for further processing, e.g. cellular scoring by apathologist or via automation.

Quality Assessment Module

Another aspect of the present disclosure is an offline mode (henceruntime is not an issue) where given a whole slide scan, individualtiles of a pre-defined size are considered, and focus score, andred-vs-black separation scores, are computed based on an algorithm whichcan suggest to the pathologist which tiles are of “better quality” andhence are “better for scoring”. This disclosure provides methods ofcomputing the focus score and red-vs-black separation score per tile,along with the experiments behind ground truth creation, and obtainingthe correlation between ground truth and computed quality metrics inorder to identify the most useful quality metrics.

Therefore, in another aspect of the present disclosure is a method forautomated quality evaluation of image tiles derived from a whole slidescan or an area of interest within a whole slide scan comprising (a)retrieving a series of digital image tiles, each digital image tilecorresponding to a particular pixel area derived from a whole slidescan, wherein the digital images are captured from a tissue specimen;(b) computing focus metrics and color separation metrics for eachdigital image, wherein the focus metrics are derived from focus featuresin each image and the color separation metrics are derived from colorseparation features in each image; and (c) identifying those digitalimage tiles more suitable for downstream processing (i.e. those that arebelieved to be better quality image tiles and thus easier to reviewand/or score) based on the focus metrics and the color separationmetrics. In some embodiments, the method further comprises generation ofa heat map or an overlay (see FIGS. 5A and 5B), which may besuperimposed over the whole slide scan or area of interest. Each ofthese steps will be described in further detail herein.

In some embodiments, the quality assessment module is run only for thosetiles of a whole slide scan that have “stronger” foreground regions. Forexample, an area of interest determine may be performed to detectsignificant foreground regions in a whole slide image (e.g. in a lowresolution image) and it is believed that this area of interestdetermination may expedite processing of the quality assessment module(which may comprise a high computing cost).

Following the identification of the significant tissue regions on theslide, each valid tissue region is divided into tiles of a given size.The tile size may be a predetermined size or may be based on an inputprovided by a user. Tile size selection is believed to be an importantprocessing step since if too large an area is captured, the tile maycontain both good quality data and bad quality data. On the other hand,if too small an area is captured, there may exist too little data (e.g.too few dots) from which to reliably estimate metrics and, eventually, aquality score. In some embodiments, the tile is sized about 300× about300 pixels, or an area of about 90,000 square pixels. In otherembodiments, the tile may be an area sized from about 40,000 squarepixels to about 160,000 square pixels.

Once the area of interest is determined and an appropriate tile size isselected, a computer system then analyzes the image tiles and featurescontained therein to compute a plurality of focus metrics and colorseparation metrics. From the plurality of focus and color separationmetrics, a focus quality score and a color separation quality score maybe computed for each tile. In some embodiments, the metrics aremulti-dimensional focus features and multi-dimensional color separationfeatures. The focus quality and color separation quality scores are thenused to provide guidance to a medical professional (e.g. a pathologist)in determining which field of view (FOV) to select for downstreamcellular scoring.

With regard reference to FIG. 3, in another aspect of the presentdisclosure is a computer-based system for determining particular tilesof a whole slide scan or particular areas of interest within a wholeslide scan which may be better suited for scoring by a medicalprofessional comprising a memory for storing images from the varioustiles; and a processor that is configured to execute a sequence ofinstructions. In general, the method involves explicit dot detection anddot classification, followed by measurement of the separation betweendifferent signals in the stained tissue sample (e.g. differently coloreddots) in a specific color space. Instructions are provided to the systemto analyze the image tiles (310) and compute (i) a plurality of focusfeatures (320), and (ii) a plurality of color separation features (330).From the plurality of focus features and the plurality of colorseparation features, focus and color separation quality scores may becomputed (340). The computed quality scores may then be used to generatea visual representation of those tiles of a whole slide scan or an areaof interest that may be more scoreable by a medical professional (step350).

Focus Features

In some embodiments, dot locations may be determined by applyingdifference of Gaussian (DoG) filters. It is believed that the peaks ofDoG correspond with likely dot locations and thus serves as a usefulfilter for detecting dot locations. Since the peak locations of the DoGfiltered image correspond to dots, and more easily identifiable dots(e.g. those with better focus) correspond to higher magnitude in the DoGfiltered image, the strength of the peaks of the DoG filtered image andstatistics derived therefrom are used as focus features herein. Examplesof DoG filter based features include:

(1) mean/median of the 0th DoG filter (coarsest scale) for all dotpixels;

(2) mean/median of the 0th DoG filter for all the black pixels;

(3) mean/median of the max DoG values of the 0th filter for all the dotblobs; and

(4) mean of the top 50% for the DoG values for the 0th filter for allthe dot blobs.

The intuition behind coming up with these statistical features is tocombine the information contained in all the dot locations in a DoGfiltered image to a few meaningful terms—hence features such asmean/median of the DoG strength (or “DoG value”) at all the estimateddot locations are used.

It has been empirically observed that using radial symmetry is also veryeffective in computing dot locations, and is a strong indicator offocus. (see Parvin, Bahram, et al. “Iterative voting for inference ofstructural saliency and characterization of subcellular events.” ImageProcessing, IEEE Transactions on 16.3 (2007): 615-623, the disclosure ofwhich is incorporated by reference in its entirety herein). In someembodiments, the above features based on mean/median based statisticscomputed on the DoG filtered image can be replaced by mean/median basedstatistics based on radial symmetry voting on the gradient magnitudeimage computed on the absorbance channel. The radial symmetry voting canbe done using minimum and maximum radius values appropriate for the dotsizes commonly seen in Dual ISH images. In some embodiments, examplevalues for radial symmetry voting include a minimum radius value of 1, amaximum radius value of 4, where the angle of the conical section usedfor radial symmetry voting can be set to pi/4 radians, and the pixelsconsidered for radial symmetry voting can have gradient magnitude valueexceeding 0.1. In some embodiments, for non-maximal suppression afterradial symmetry voting, a neighborhood of about 2 pixels is used.

The skilled artisan will appreciate that these features may be computedin any sequence. The skilled artisan will also appreciate that theparameters of the filters, e.g. radii values, may be altered such as tocapture dots of different sizes.

Color Separation Features

In some embodiments, color separation features are computed by runningvarious color separation filters to distinguish differently coloreddots. For example, color separation features may be computed todistinguish between black dots and red dots (and vice versa). In someembodiments, color deconvolution is performed on the collected imagesbased on the various constituent color vectors (in the specific Dual ISHexample exemplified here, the main constituent colors are black and red,for the HER2 and Chromosome 17 gene expressions, respectively and therecan be falsely identified dot pixels which are blue in color as theybelong to the cell nucleus stained with haematoxylin). Using the colorinformation, the system may classify dot pixels into red dots and blackpixels, or blue (if it is a falsely identified dot pixel).

Color deconvolution is a method where for a given image pixel, thecontributions of the different stain vectors, that combine together tocreate that single pixel, are computed using as input the knowledge ofthe ground truth stain vectors. (See, Ruifrok, Arnout C., and Dennis A.Johnston. “Quantification of histochemical staining by colordeconvolution.” Analytical and quantitative cytology and histology/theInternational Academy of Cytology [and] American Society of Cytology23.4 (2001): 291-299, the disclosure of which is incorporated byreference in its entirety). Color deconvolution (“unmixing”) isdescribed further herein. In some embodiments, the ground truth colorcomponents typically correspond to the colors of the individual stainsor probes to which the tissue sample was treated. As such, the new colorcomponents span a new color space that, in some embodiments, allows forviewing of a single color (e.g. red for red dots) at a time.

In some embodiments, to detect color separation in an image, the atleast some of the following features are determined:

(1) Amax (max of the smoothened version of A channel, where A iscomputed from RGB to LAB conversion);

(2) Asigmax (max of Asig, the sigmoid enhanced version of the smoothenedversion of A channel);

(3) a maximum in unmixed black channel (considering the unmixed blackchannel obtained after color deconvolution);

(4) a gradient value for various color channels—luminance (L), unmixedred (unmixed red channel obtained after color deconvolution), A channel(in L-A-B color space), Asig channel (sigmoid enhanced version of Achannel), green channel;

(5) the DoG values in (i) the green channel, (ii) A channel, and (iii)optical density domain absorbance channel (given a RGB pixel, itsoptical density representation is (ROD, GOD, BOD) as shown in Eq 2; thenabsorbance OD=sqrt(ROD*ROD+GOD*GOD+BOD*BOD); and

(6) a color domain features, based on modeling the color information (inoptical density color space) in terms of ellipses, where the ellipsesare then fitted to different colored pixels (e.g. black and red dotpixels) separately, and then certain features are computed based on howseparated the different colored ellipses are from each other (e.g. howdifferent a black ellipse is from a red ellipse). In general, thefeatures computed from color domain separation ellipses include (a) meanand standard deviation for ellipses fitted to differently colored dots(e.g. black and red colored dots); (b) the distance between differentlycolored ellipses (e.g. between black and red ellipses); and (c) thedimensions of color ellipses. An example of ellipse fitting to red andblack dots is described in Example 4 herein. Specific features derivedfrom ellipse fitting, which may be used in computing the various metricsneeded for quality score determination, are FIG. 8, wherein the topgraph shows comparatively better dot separation (as compared with thelower graph) resulting in a comparatively better discrimination betweenthe black dots and the red dots in the cellular images.

Once the various focus features and color separation features have beencomputed, the computer executes instructions to compute focus qualityscores and color separation quality scores for all image tilesclassified as having at least one of each differently colored dot(again, the dots corresponding to different in situ hybridizationsignals in the images derived from the tissue sample). For example, andin the context of Dual ISH, focus scores and color separation scores arecomputed only for those tiles having at least one black dot and at leastone red dot. The reason behind this is that according to the Dual ISHscoring guidelines, a trained pathologist considers only those cells forscoring which have at least one black dot and one red dot.

In some embodiments, instructions are provided to combine the variousfocus features (e.g. multi-dimensional focus features) into a singlefocus feature value. Likewise, the various color separation features(e.g. multi-dimensional color separation features) are combined into asingle color separation feature value.

In some embodiments, the various focus and various color separationfeatures, respectively, are combined using a generalized linearregression algorithm, as known in the art. Generalized linear regressionis a modification of the ordinary linear regression that allows errordistribution models other than that provided by a conventional normaldistribution for linear fitting of data with ground truth. As such,given “n” d-dimensional vectors (x1, x2, x3, . . . ,xn), with “n”corresponding ground truth scores (g1, g2 . . . gn), where xi={xi1, xi2,. . . , xid}, instructions are provided to compute a d-dimensionalweight vector w=(w1 w2 . . . wd), such that an inner product of each xkwith w will be similar to gk. This computation may be achieved, forexample, with “glmfit” in Matlab where normal distribution is used tomodel the fitting error. (See, for example, Dobson, A. J. AnIntroduction to Generalized Linear Models. New York: Chapman & Hall,1990; McCullagh, P., and J. A. Nelder. Generalized Linear Models. NewYork: Chapman & Hall, 1990; and Collett, D. Modeling Binary Data. NewYork: Chapman & Hall, 2002). Examples of ground truths used as input inthis process are exemplified in Examples 2 and 3.

Two weight vectors are derived from the linear regression. A firstweight vector is computed based on the multi-dimensional focus features(WF); and a second weight vector is computed based on themulti-dimensional color separation features (WRB). Using, for example,“glmfit” on the multi-dimensional focus features and studying itscorrespondence with the ground truth inputs for focus, the inner productbetween WF and a focus feature allows the computation of a focus qualityscore for a given focus feature. Similarly, using “glmfit” on themulti-dimension color separation features and studying itscorrespondence with the ground truth inputs of color separation, theinner product between WRB and a color separation feature allows thecomputation of a color separation quality score for a color separationindicating feature.

Once the focus score and color separation score are computed for eachimage tile, instructions are executed to return a pre-determined or userdefined number of regions which may represent the better quality regionsfor the given image. This is demonstrated in Example 7 herein.

In some embodiments, the better quality regions may be displayedvisually to a medical professional. In some embodiments, the visualrepresentation may be in the form of a “heat map.” In some embodiments,the heat map may be color coded, e.g. warm colors like red or orange andcool colors like blue or purple to delineate the range of quality tiles.Advantageously, a “heat map” is generated wherein the pseudo colors,applied to positions on a map, correspond spatially to the image, andthe colors are chosen to represent quality assessments of the tiles. Aheat map may be generated, in some embodiments, by applying a low passfilter to an image of a whole tissue slide or an area of interest. Thegeneration of heat maps of marker expression are more fully described inPCT/EP/2015/062015, entitled “An Image Processing Method and System forAnalyzing a Multi-Channel Image Obtained From a Biological Tissue SampleBeing Stained By Multiple Stains,” the disclosure of which is herebyincorporated by reference herein in its entirety.

In some embodiments, instructions may be provided to generate anannotation file (e.g. xml), where the annotation file is used to rendera bounding box around each valid tissue tile. In some embodiments, theannotation file also includes other textual information (e.g. focusscore, color separation score, and the number of differently coloredpixels). The color of the annotation xml per tile indicates how useful atile is, such as by an ease-of-scoring point of view. An example isshown in FIGS. 5A and 5B. For example, a tile may be “black” in theannotation xml file if the region is not of scoreable quality, and maybe colored “white” if the region is of “good quality” and is “wellscoreable.” Of course, any color may be assigned to any tile provided akey or legend is provided to give meaning to each color.

When the medical professional examines a whole slide image, theprofessional assumes the responsibility of selecting a proper field ofview (FOV) from where 20 cells may be selected for downstream scoring.To automate the process, the aforementioned steps may be carried out toreturn quality scores for each individual tile (where the tile size ispre-determined) and then, based on the quality scores and the number ofdifferently colored dots returned per tile, descriptors such as“good/bad/in-between” may be assigned to all the tiles. Normally, if aquality metric (focus or color separation) is reasonable enough, anygenerated heat map may help the medical professional consider the more“scoreable” area (assigned high scores by the quality metrics), thusassisting the medical professional in selecting good FOVs for downstreamcell based scoring. Thus, it is believed, that the generation of a heatmap is not intended to replace the medical professional in FOVselection, but to provide guidance to the medical professional in FOVselection, therefore allowing the task to be simpler, faster, and/ormore accurate. In FIG. 5A, the heat map is superimposed on a lowresolution image, where white tile borders (“white tiles”) denote higherquality regions and black tile borders (“black tiles”) denote poorerquality regions. Intermediate quality tiles may be shown in variousshades of grey and respectively denote various intermediate levels ofquality. FIG. 5B provides an exploded view of a portion of FIG. 5A, withwhite tiles, black tiles, and grey shaded tile borders. Any number ofdifferently shaded borders may be used in the heat map, as well asdifferent colors and shadings of colors. In some embodiments, along withthe tile shading and/or color, there is textual information associatedwith each tile that captures information, including focus score,red-vs-black color separation score, number of red dots in the tile, andnumber of black dots in the tile.

Any of the images evaluated by the computer system may be pre-processedprior to any identification of features, computation of metrics, orevaluation by the computer system. For example, noise reduction filtersand other enhancing filters may be run as known to those of ordinaryskill in the art. The intuition behind using noise reduction filters isto remove small objects of discontinuity (e.g. small areas of specklingand dust, can be removed by median filtering, and hence spurious dotswill not picked up in these noisy regions; similarly, inside a cellwhich contains red and black dots, DoG filtering can enhance the visualdistinctiveness of the dots and make them easier to pick up). Theprocesses disclosed herein may also be repeated any number of times toevaluate any desired areas of interest.

The computer system of the present disclosure may be tied to a specimenprocessing apparatus which can perform one or more preparation processeson the tissue specimen. The preparation process can include, withoutlimitation, deparaffinizing a specimen, conditioning a specimen (e.g.,cell conditioning), staining a specimen, performing antigen retrieval,performing immunohistochemistry staining (including labeling) or otherreactions, and/or performing in situ hybridization (e.g., SISH, FISH,etc.) staining (including labeling) or other reactions, as well as otherprocesses for preparing specimens for microscopy, microanalyses, massspectrometric methods, or other analytical methods.

A specimen can include a tissue sample. The sample of tissue can be anyliquid, semi-solid or solid substance (or material) in or on which atarget can be present. In particular, a tissue sample can be abiological sample or a tissue sample obtained from a biological tissue.The tissue can be a collection of interconnected cells that perform asimilar function within an organism. In some examples, the biologicalsample is obtained from an animal subject, such as a human subject. Abiological sample can be any solid or fluid sample obtained from,excreted by or secreted by any living organism, including withoutlimitation, single celled organisms, such as bacteria, yeast,protozoans, and amoebas among others, multicellular organisms (such asplants or animals, including samples from a healthy or apparentlyhealthy human subject or a human patient affected by a condition ordisease to be diagnosed or investigated, such as cancer). For example, abiological sample can be a biological fluid obtained from, for example,blood, plasma, serum, urine, bile, ascites, saliva, cerebrospinal fluid,aqueous or vitreous humor, or any bodily secretion, a transudate, anexudate (for example, fluid obtained from an abscess or any other siteof infection or inflammation), or fluid obtained from a joint (forexample, a normal joint or a joint affected by disease). A biologicalsample can also be a sample obtained from any organ or tissue (includinga biopsy or autopsy specimen, such as a tumor biopsy) or can include acell (whether a primary cell or cultured cell) or medium conditioned byany cell, tissue or organ. In some examples, a biological sample is anuclear extract. In certain examples, a sample is a quality controlsample, such as one of the disclosed cell pellet section samples. Inother examples, a sample is a test sample. For example, a test sample isa cell, a tissue or cell pellet section prepared from a biologicalsample obtained from a subject. In an example, the subject is one thatis at risk or has acquired a particular condition or disease. In someembodiments, the specimen is breast tissue.

The processing apparatus can apply fixatives to the specimen. Fixativescan include cross-linking agents (such as aldehydes, e.g., formaldehyde,paraformaldehyde, and glutaraldehyde, as well as non-aldehydecross-linking agents), oxidizing agents (e.g., metallic ions andcomplexes, such as osmium tetroxide and chromic acid),protein-denaturing agents (e.g., acetic acid, methanol, and ethanol),fixatives of unknown mechanism (e.g., mercuric chloride, acetone, andpicric acid), combination reagents (e.g., Carnoy's fixative, methacarn,Bouin's fluid, B5 fixative, Rossman's fluid, and Gendre's fluid),microwaves, and miscellaneous fixatives (e.g., excluded volume fixationand vapor fixation).

If the specimen is a sample embedded in paraffin, the sample can bedeparaffinized using appropriate deparaffinizing fluid(s). After thewaste remover removes the deparaffinizing fluid(s), any number ofsubstances can be successively applied to the specimen. The substancescan be for pretreatment (e.g., protein-crosslinking, expose nucleicacids, etc.), denaturation, hybridization, washing (e.g., stringencywash), detection (e.g., link a visual or marker molecule to a probe),amplifying (e.g., amplifying proteins, genes, etc.), counterstaining,coverslipping, or the like.

The specimen processing apparatus can apply a wide range of substancesto the specimen. The substances include, without limitation, stains,probes, reagents, rinses, and/or conditioners. The substances can befluids (e.g., gases, liquids, or gas/liquid mixtures), or the like. Thefluids can be solvents (e.g., polar solvents, non-polar solvents, etc.),solutions (e.g., aqueous solutions or other types of solutions), or thelike. Reagents can include, without limitation, stains, wetting agents,antibodies (e.g., monoclonal antibodies, polyclonal antibodies, etc.),antigen recovering fluids (e.g., aqueous- or nonaqueous-based antigenretrieval solutions, antigen recovering buffers, etc.), or the like.

Probes can be an isolated nucleic acid or an isolated syntheticoligonucleotide, attached to a detectable label or reporter molecule.Labels can include radioactive isotopes, enzyme substrates, co-factors,ligands, chemiluminescent or fluorescent agents, haptens, and enzymes.For example, probes can include, without limitation, a hapten-labeledspecific binding moiety, a DNA probe (e.g., DNP-labeled DNA probe), anitroaryl compound, dinitrophenol, an electron-deficient aromaticcompound, a probe hybridization solution, or other type of ISH probes.ISH can involve a labeled complementary DNA or RNA strand (i.e., probe)to localize a specific DNA or RNA sequence in a portion or section oftissue (in situ), or, if the tissue is small enough, in the entiretissue (whole mount ISH).

In some embodiments, a cocktail assay applied by the processingapparatus includes different reagents. For example, one cocktail assayincludes the ULTRAVIEW SISH Detection Kit (Ventana Medical Systems,Inc., p/n 780-001), the INFORM HER2 DNA Probe (Ventana Medical Systems,Inc., p/n 780-4332), the Rabbit Anti-DNP Antibody (Ventana MedicalSystems, Inc., p/n 780-4335), the Rabbit Anti-HER2 (4B5) Antibody(Ventana Medical Systems, Inc., p/n 800-2996), the ULTRAVIEW UniversalAlkaline Phosphatase Red Detection Kit (Ventana Medical Systems, Inc.,p/n 760-501), the silver wash (Ventana Medical Systems, Inc., p/n780-002), and/or the INFORM Chromosome 17 Probe (Ventana MedicalSystems, Inc., p/n 780-4331). Another cocktail assay is the INFORM HER2Dual ISH DNA Probe sold by (Ventana Medical Systems, Inc.), whichincludes the INFORM HER2 Dual ISH DNA Probe Cocktail (Ventana MedicalSystems, Inc., p/n 800-4422), the HybReady (Ventana Medical Systems,Inc., p/n 780-4409), the ultraView SISH DNP Detection Kit (VentanaMedical Systems, Inc., p/n 800-098), the ultraView Red ISH DIG DetectionKit (Ventana Medical Systems, Inc., p/n 800-505), the ultraView SilerWash II (Ventana Medical Systems, Inc., p/n 780-003), and/or the HER2Dual ISH 3-in-1 Xenograft Slides (Ventana Medical Systems, Inc., p/n783-4332). Other cocktail assays can be used. Cocktail assays can beused to quantitatively detect amplification of the HER2 gene via twocolor chromogenic ISH in formalin-fixed, paraffin-embedded tissuespecimens of human breast cancer and gastric cancer, including thegastro-esophageal junction, and can be an aid in the assessment ofpatients for whom Herceptin (trastuzumab) may be a treatment option. Inyet other protocols, the cocktail assay is the VENTANA HER2 DNA ProbeAssay sold by Ventana Medical Systems, Inc., p/n 800-4422. U.S. patentapplication Ser. No. 11/809,024 (corresponding to U.S. PatentPublication No. 2008/299555) entitled MULTICOLOR CHROMOGENIC DETECTIONOF BIOMAKERS and U.S. patent application Ser. No. 11/809,024(corresponding to U.S. Patent Publication No. 2011/0136130) entitledMETHOD FOR CHROMOGENIC DETECTION OF TWO OR MORE TARGET MOLECULES IN ASINGLE SAMPLE disclose substances, protocols, and specimen processingtechniques and are incorporated by reference in their entireties.

The specimen processing apparatus can be an automated apparatus, such asthe BENCHMARK XT instrument and SYMPHONY instrument sold by VentanaMedical Systems, Inc. Ventana Medical Systems, Inc. is the assignee of anumber of United States patents disclosing systems and methods forperforming automated analyses, including U.S. Pat. Nos. 5,650,327,5,654,200, 6,296,809, 6,352,861, 6,827,901 and 6,943,029, and U.S.Published Patent Application Nos. 20030211630 and 20040052685, each ofwhich is incorporated herein by reference in its entirety.Alternatively, specimens can be manually processed.

After the specimens are processed, a user can transport specimen-bearingslides to the imaging apparatus. The imaging apparatus used here is abrightfield imager slide scanner. One brightfield imager is the iScanCoreo™ brightfield scanner sold by Ventana Medical Systems, Inc. Inautomated embodiments, the imaging apparatus is a digital pathologydevice as disclosed in International Patent Application No.:PCT/US2010/002772 (Patent Publication No.: WO/2011/049608) entitledIMAGING SYSTEM AND TECHNIQUES or disclosed in U.S. Patent ApplicationPublication No. 2014/0178169, filed on Sep. 9, 2011, entitled IMAGINGSYSTEMS, CASSETTES, AND METHODS OF USING THE SAME. International PatentApplication No. PCT/US2010/002772 and U.S. Patent ApplicationPublication No. 2014/0178169 are incorporated by reference in theirentities. In other embodiments, the imaging apparatus includes a digitalcamera coupled to a microscope.

Embodiments of the subject matter and the operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments of the subject matterdescribed in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage medium for execution by, or tocontrol the operation of, data processing apparatus.

A computer storage medium can be, or can be included in, acomputer-readable storage device, a computer-readable storage substrate,a random or serial access memory array or device, or a combination ofone or more of them. Moreover, while a computer storage medium is not apropagated signal, a computer storage medium can be a source ordestination of computer program instructions encoded in an artificiallygenerated propagated signal. The computer storage medium can also be, orcan be included in, one or more separate physical components or media(e.g., multiple CDs, disks, or other storage devices). The operationsdescribed in this specification can be implemented as operationsperformed by a data processing apparatus on data stored on one or morecomputer-readable storage devices or received from other sources.

The term “programmed processor” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable microprocessor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing. The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus also can include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,subprograms, or portions of code). A computer program can be deployed tobe executed on one computer or on multiple computers that are located atone site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., an LCD (liquid crystal display), LED(light emitting diode) display, or OLED (organic light emitting diode)display, for displaying information to the user and a keyboard and apointing device, e.g., a mouse or a trackball, by which the user canprovide input to the computer. In some implementations, a touch screencan be used to display information and receive input from a user. Otherkinds of devices can be used to provide for interaction with a user aswell; for example, feedback provided to the user can be in any form ofsensory feedback, e.g., visual feedback, auditory feedback, or tactilefeedback; and input from the user can be received in any form, includingacoustic, speech, or tactile input. In addition, a computer can interactwith a user by sending documents to and receiving documents from adevice that is used by the user; for example, by sending web pages to aweb browser on a user's client device in response to requests receivedfrom the web browser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), an inter-network (e.g., the Internet), andpeer-to-peer networks (e.g., ad hoc peer-to-peer networks). For example,the network 20 of FIG. 1 can include one or more local area networks.

The computing system can include any number of clients and servers. Aclient and server are generally remote from each other and typicallyinteract through a communication network. The relationship of client andserver arises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data (e.g., an HTML page) to a clientdevice (e.g., for purposes of displaying data to and receiving userinput from a user interacting with the client device). Data generated atthe client device (e.g., a result of the user interaction) can bereceived from the client device at the server.

It is to be noted that for Coreo and HT scanners, the green channel wasused to compute focus metrics for scanning in general, and not forscanning for Dual ISH slides in particular; in this patent application,focus metrics have been proposed for Dual ISH scans in particular.

Methods of unmixing are well known to those of ordinary skill in the artand any method now known or later discovered may be used to “unmix”multiplex images into the vessel channel images. In general, theunmixing process extracts stain-specific channels to determine localconcentrations of individual stains using reference spectra that arewell known for standard types of tissue and stain combinations. Forexample, each pixel in an input image may comprise a mixture ofcomponent spectra including one or more quantum dots representing targetstructures, in addition to broadband signals such as DAPI andautofluorescence, as described above. The unmixing may use referencespectra retrieved from a control image or estimated from the image underobservation. Unmixing the component signals of each input pixel enablesretrieval and analysis of stain-specific channels, such as vesselchannels and nuclei channels. The terms “unmixing” and “colordeconvolution” (or “deconvolution”) or the like (e.g. “deconvolving,”“unmixed”) are used interchangeably in the art.

In some embodiments, the multiplex images are unmixed using linerunmixing. Linear unmixing is described, for example, in ‘Zimmermann“Spectral Imaging and Linear Unmixing in Light Microscopy” Adv BiochemEngin/Biotechnol (2005) 95:245-265’ and in in C. L. Lawson and R. J.Hanson, “Solving least squares Problems”, PrenticeHall, 1974, Chapter23, p. 161,’ the disclosures of which are incorporated herein byreference in their entirety. In linear stain unmixing, the measuredspectrum (S(λ)) at any pixel is considered a linear mixture of stainspectral components and equals the sum of the proportions or weights (A)of each individual fluorophore reference spectral signature (R(λ)) thatis being expressed at the pixelS(λ)=A1·R1(λ)+A2·R2(λ)+A3(λ) . . . Ai·Ri(λ)

which can be more generally expressed as in matrix form asS(λ)=ΣAi·Ri(λ) or S=R·A

If there are M channels images acquired and N individual fluorophores,the columns of the M×N matrix R is the known reference spectralsignature of the individual fluorophores and the N×1 vector A is theunknown of the proportions of individual fluorophores and the M×1 vectorS is the measured multichannel spectral vector at a pixel. In theseequations, the signal in each pixel (S) is measured during acquisitionof the multiplex image and the reference spectra for the known stainsare usually determined in an independent offline method from fluorescentspecimens labeled with only a single stain using identical instrumentsettings. It becomes a simple linear algebra matrix exercise todetermine the contributions of various stains (Ai) by calculating theircontribution to each point in the measured spectrum. In someembodiments, the solution is obtained using an inverse least squaresfitting approach that minimizes the square difference between themeasured and calculated spectra by solving the the following set ofequations,[∂Σj {S(λj)−ΣAi·Ri(λj)}2]/∂Ai=0

In this equation, j represents the number of detection channels and iequals the number of stains. The linear equation solution often involvesallowing a constrained unmixing to force the weights (A) to sum tounity.

In other embodiments, unmixing is accomplished using the methodsdescribed in WO2014/195193, entitled “Image Adaptive PhysiologicallyPlausible Color Separation,” filed on May 28, 2014, the disclosure ofwhich is hereby incorporated by reference in its entirety herein. Ingeneral, WO2014/195193 describes a method of unmixing by separatingcomponent signals of the input image using iteratively optimizedreference vectors. In some embodiments, image data from an assay iscorrelated with expected or ideal results specific to thecharacteristics of the assay to determine a quality metric. In the caseof low quality images or poor correlations against ideal results, one ormore reference column vectors in matrix R are adjusted, and the unmixingis repeated iteratively using adjusted reference vectors, until thecorrelation shows a good quality image that matches physiological andanatomical requirements. The anatomical, physiological, and assayinformation may be used to define rules that are applied to the measuredimage data to determine the quality metric. This information includeshow the tissue was stained, what structures within the tissue wereintended or not intended to be stained, and relationships betweenstructures, stains, and markers specific to the assay being processed.An iterative process results in stain-specific vectors that can generateimages that accurately identify structures of interest and biologicallyrelevant information, are free from any noisy or unwanted spectra, andtherefore fit for analysis. The reference vectors are adjusted to withina search space. The search space defines a range of values that areference vector can take to represent a stain. The search space may bedetermined by scanning a variety of representative training assaysincluding known or commonly occurring problems, and determininghigh-quality sets of reference vectors for the training assays.

In other embodiments, unmixing is accomplished using the methodsdescribed in WO2015/124772, entitled “Group Sparsity Model for ImageUnmixing,” filed on Feb. 23, 215, the disclosure of which is herebyincorporated by reference in its entirety herein. In general,WO2015/124772 describes unmixing using a group sparsity framework, inwhich fractions of stain contributions from a plurality of colocationmarkers are modeled within a “same group” and fractions of staincontributions from a plurality of non-colocation markers are modeled indifferent groups, providing co-localization information of the pluralityof colocation markers to the modeled group sparsity framework, solvingthe modeled framework using a group lasso to yield a least squaressolution within each group, wherein the least squares solutioncorresponds to the unmixing of the colocation markers, and yielding asparse solution among the groups that corresponds to the unmixing of thenon-colocation markers. Moreover, WO2015124772 describes a method ofunmixing by inputting image data obtained from the biological tissuesample, reading reference data from an electronic memory, the referencedata being descriptive of the stain color of each one of the multiplestains, reading colocation data from electronic memory, the colocationdata being descriptive of groups of the stains, each group comprisingstains that can be collocated in the biological tissue sample, and eachgroup forming a group for the group lasso criterion, at least one of thegroups having a size of two or above, and calculating a solution of thegroup lasso criterion for obtaining the unmixed image using thereference data as a reference matrix. In some embodiments, the methodfor unmixing an image may comprise generating a group sparsity modelwherein a fraction of a stain contribution from colocalized markers isassigned within a single group and a fraction of a stain contributionfrom non-colocalized markers is assigned within separate groups, andsolving the group sparsity model using an unmixing algorithm to yield aleast squares solution within each group.

In a further aspect, the disclosure relates to an image analysis methodfor determining one of a plurality of z-layers in a z-stack of digitalimages, whereby the image of the determined z-layer is to be used forcomputing tissue-related or cell-related scores. The z-stack of digitalimages depict different layers of a single tissue sample located on aslide. The z-stack images have been acquired by varying the focus of anoptical sensor, e.g. the sensor of a slide scanning system. The methodis implemented by an image analysis system and comprises:

-   -   for each of the z-layers, computing focus metrics. Each focus        metric is an indicator of focus quality of the z-layer and is        derived from image data of said z-layer;    -   for each of the z-layers, computing color separation metrics.        Each color separation metric is an indicator of color separation        quality of the z-layer and being derived from image data of said        z-layer; and    -   evaluating the focus metrics and the color separation metrics        and determining the one of the z-layers to be used for computing        tissue-related or cell-related scores in dependence on the focus        metrics and color separation metrics of each of the z-layers.

As used herein, “z” corresponds to a third dimension in space, wherein“x” and “y” correspond to first and second dimensions in space whichcorrespond to the axes of the digital images, wherein x, y and z axesare orthogonal to each other. The units of the x and y axis of an imagemay be given in number of pixels while the unit of the z axis of thez-stack may be given as the index or number of a respective z-layer. Forexample, the z-stack may comprise 15 layers of images having beenobtained from the same tissue sample on a particular slide, whereby eachof said 15 images may correspond to a different focus setting of theoptic sensor having been used for generating the z-stack of images.

Said features may be advantageous as the combination of focus score andcolor separation score may provide for a more robust and accurate way ofidentifying an image layer and corresponding image that is in-focus andin addition allows distinguishing different tissue features based oncolor separation. This may allow for better and more accurate resultswhen performing downstream image processing, e.g. cell detection and dotclassification. In a further advantageous aspects, at least some focusmetrics and color separation metrics (in particular those not beingbased on the analysis of individual dots and tiles) can be computed veryquickly, e.g. in real time, allowing the identification of a highquality z-layer image while the image scanning process is still goingon.

According to embodiments, the method comprises analyzing selectively theone of the images in the z-stack of digital images that corresponds tothe determined z-layer and selectively computing the tissue-related orcell-related scores for said analyzed image. This may result in a higheraccuracy of the computed scores and may result in the reduction of CPUconsumption as score computation is not performed for multiple layers ofthe z-stack but rather for the one image of the z-stack having the bestcombination of focus and color separation quality.

According to embodiments, the evaluation of the focus metrics and thecolor separation metrics comprises:

-   -   computing the difference between the z-layer having best focus        and the z-layer having best color separation by computing an        absolute value of the difference of the index position of the        z-layer having best focus and the index position of the z-layer        having best color separation in the z-stack; for example, in a        z-stack of 15 layers, the best focus layer may have stack index        9 and the best color separation layer may have stack index 5. In        this example, the difference of the z-layer having best focus        and the z-layer having best color separation would be 9−5=4.    -   determining whether the computed difference is greater than,        equal to, or less than a predetermined threshold value;    -   in case the computed difference is determined to be less than or        equal to the predetermined threshold value: computing an average        of the z-layer index of the z-layer having best focus and the        z-layer index of the z-layer having best color separation, if        the average of the z-layer index is an integer, identifying the        one of the z-layers of the z-stack whose index position is        identical to the computed average of the z-layer indices and        using said one identified z-layer as the one z-layer to be used        for computing the scores; in the above mentioned example, the        average of the two z-layer indices would be (9+5)/2=7. In this        case, the image at z-layer index=7 would be the one z-layer        identified. If the average of the z-layer index is not an        integer, identifying the two consecutive z-layers of the z-stack        which lie next to the computed average of the z-layer indices;        identifying the one of the two consecutive z-layers having the        highest color separation metrics; and using said one identified        z-layer as the one z-layer to be used for computing the scores.        For example, the best focus and best color separation layers may        have the z-index 9 and 6, the average of the two z-layer indices        would be (9+6)/2=7.5. In this case, the two consecutive z-layers        7 and 8 would be identified. If for the image of z-layer 7 a        higher color separation metrics was computed than for the image        of z-layer 8, then the 7^(th) layer is the one selected layer.        Otherwise, the 8^(th) layer is the one selected layer.

According to embodiments, in case the computed difference is determinedto be greater than the predetermined threshold value, the methodcomprises:

-   -   comparing a focus score derived from the focus metrics of the        z-layer image having best focus with a reference focus score for        determining a deviation of the focus score and the reference        focus score; the reference focus score is an empirically        determined focus score obtained for a digital image (which may        also be referred to as training digital image) depicting the        same type of tissue sample whereby said digital image has been        manually identified as being in-focus. For example, using the        same type of tissues sample for computing a focus metrics and a        reference focus score means that the tissue types for which the        focus metrics and the reference focus score were computed are        the same or similar, comprise the same or similar composition of        biomarkers and were stained with the same staining protocol, the        respective digital images having been acquired by the same or        similar type of method and apparatus. The reference focus score        is obtained before the z-stack of images is acquired. For        example, a ground truth data set, also known as training data        set or learning data set, may comprise a plurality of z-stacks        of training digital images, each z-stack of training digital        images depicting multiple layers of a respective training tissue        sample. Focus metrics may be computed for each z-layer image and        each of the training z-stacks of images. From one or more of the        training digital images (of one or more of the z-stacks of        training images) having been manually identified as in-focus        images, the reference focus score is computed;    -   comparing a color separation score derived from the color        separation metrics of the z-layer image having best color        separation with a reference color separation score for        determining a deviation of the color separation score and the        reference color separation score, the reference color separation        score being an empirically determined color separation score        obtained for a digital image (which may also be referred to as        training digital image) depicting the same type of tissue sample        and having been manually identified as being a high quality        color separation image; The pre-determination or pre-computation        of the reference color separation score may be performed        analogously as described for the reference focus score;    -   in case the deviation of the focus score from the reference        focus score is greater than the deviation of the color        separation score from the reference color separation score,        using the z-layer image having best color separation for        computing the scores.

For example, this comparison may involve a normalization of the focusscores and the color separation scores to make the score deviationscomparable. For example, a normalized focus score NFS may be computedfrom a focus score FS computed for a particular z-layer image and froman average expected focus score AEFS according to the formulaNFS=FS/AEFS. The average expected focus score AEFS may have beenobtained empirically from averaged focus scores computed for a largenumber of in-focus and out-of-focus images. Likewise, a normalized colorseparation score NCSS may be computed from a color separation score CSScomputed for a particular z-layer image and from an average expectedfocus score ACSS according to the formula NCSS=CSS/ACSS. The averageexpected focus score ACSS may have been obtained empirically fromaveraged focus scores computed for a large number of in-focus andout-of-focus images.

According to embodiments, in case the deviation of the color separationscore from the reference color separation score is greater than thedeviation of the focus score from the reference focus score, the methodcomprises using the z-layer image having best focus for computing thescores. As explained above, the focus scores and color separation scoresmay be normalized for making them comparable.

According to embodiments, the method further comprises:

-   -   computing, for each of the z-layers, a focus score by using the        focus metrics of the digital image corresponding to the z-layer        as input;    -   computing, for each of the z-layers, a color separation score by        using the color separation metrics of the digital image        corresponding to the z-layer as input;    -   identifying the z-layer having best focus and identifying the        z-layer having best color separation. Said two z-layers are        identified independently from each other.

According to embodiments, the identification of the z-layer having bestfocus comprises:

-   -   storing the focus score of each of the z-layers in a respective        element of a focus score vector;    -   vector element-wise moving a smoothing-window across the focus        score vector, whereby at each moving step of the smoothing        window, an average or median of the focus scores in the focus        score vector elements covered by the smoothing window are        computed; typically, the smoothing-window covers at least three        elements of the score vector. In other embodiments, the        smoothing window may cover 4, 5 or even more elements of the        focus score vector. If the number of z-layers and corresponding        elements of the score vector is very high, the window may be        larger and cover e.g. 5 or more elements;    -   generating a smoothed focus score vector whose elements comprise        the averaged or median focus scores instead of the original        focus scores of the focus vector; the elements at the beginning        and end of the smoothed vector may still have the original        values. Typically, the average or median is assigned to the        vector element lying under the center of the sliding window;    -   identifying the one of the z-layers that corresponds to the        element in the smoothed focus score vector comprising the        maximum focus score value as the z-layer having highest focus        score. The smoothing of the score values may increase accuracy        of selecting the one z-layer as score outliers may be reduced.

According to embodiments, the identification of the z-layer having bestcolor separation comprises:

-   -   storing the color separation score of each of the z-layers in a        respective element of a color separation score vector;    -   vector element-wise moving a smoothing-window across the color        separation vector, whereby at each moving step of the smoothing        window, an average or median of the color separation scores in        the color separation score vector elements covered by the        smoothing window are computed; the size of the smoothing window        for the color separation score vector may be the same as        described for the smoothing window of the focus score vector;    -   generating a smoothed color separation score vector whose        elements comprise the averaged or median color separation scores        instead of the original color separation scores of the color        separation vector;    -   identifying the one of the z-layers that corresponds to the        element in the smoothed color separation score vector comprising        the maximum color separation score value as the z-layer having        highest color separation score.

According to embodiments, the tissue sample has been stained at leastwith a first stain having a first color and with a second stain having asecond color; the first and second colors differ from each other. Thecolor separation metric may be an indicator of color separation qualityof the first and the second color. The first and/or second stain may befor example a fluorescence stain, silver stain, hematoxylin stain or anyother kind of stain used for staining tissue samples, cells, cellcomponents and/or biomarkers. The “color” here means the light spectrumemitted from a particular stain irrespective of the underlying physicalemission process (e.g. fluorescence, reflection, etc.). For example, thefirst stain may be a silver stain and the first color may be “black”(wherein black herein refers to a characteristic light spectrum emittedfrom silver-stained sample sections). The second stain may be afluorescent stain and the second color may be red.

According to embodiments, each of the digital images is an RGB image.The computation of the color separation metrics comprises, for each ofthe digital images:

-   -   computing a derivative image from the digital RGB image by        computing a first derivative value DER1 for each of the pixels        in the digital RGB image according to DER1=R+a_(CS)*G+b_(CS)*B,        where −1≤a_(CS)<1, where −1≤b_(CS)<1, wherein R, G and B are the        red, green and blue components of the pixel intensity of said        pixel in the RGB image, wherein a_(CS) and b_(CS) are parameter        values having been empirically determined as values which, when        used for computing color separation metrics, provide color        separation metrics having predictive power in respect to color        separation quality of an image;    -   computing a color separation metric J_(CS) according to the        formula        J _(CS)=Σ_(x)Σ_(y)(|DER1(x,y)−DER1(x,y−s)|²),

wherein x and y are the x and y coordinates of the pixels in thederivative image and s is a number of pixel distance positions, wherebys has been empirically determined as pixel distance positions thatcorrelate with high J_(CS) values in images of known, high colorseparation quality. The color separation metrics may be of particularrelevance in respect to the capability to separate (e.g. outputdifferent values for) the colors of the stains used, e.g. in respect tothe first and second color. In the example case of the z-stackcomprising 15 z-layers, 15 derivative images and respective J_(CS)values are computed.

According to embodiments, two or more of the focus metrics computed fora particular z-layer image of the z-stack are computed into a singlefocus score. Likewise, two or more of the color separation metricscomputed for a particular z-layer image of the z-stack are computed intoa single color separation score. For example, this may be performed bysumming up or multiplying all the focus metrics computed for aparticular z-layer image or applying an aggregation heuristics.Analogously, the computation of the color separation score may beperformed by summing up or multiplying all the color separation metricscomputed for a particular z-layer image.

According to embodiments, the method further comprises empiricallydetermining the parameters a_(CS) and b_(CS) by:

-   -   obtaining a plurality of different candidate a-values and        candidate b-values for a_(CS) and b_(CS), each candidate a-value        and each candidate b-value lying in the range [−1; +1];    -   obtaining a plurality of training z-layers in a z-stack of        training digital images, each training digital image being an        RGB image having assigned a color separation score known to        accurately indicate the color separation quality of the training        image;    -   for each possible combination of a candidate a-value and a        candidate b-value, and for each of the training images in the        training z-stack, computing a derivative image according to the        formula DER1 _(acand-bcand)=R+a_(CS-cand)*G+b_(CS-cand)*B,        wherein R, G and B are the red, green and blue components of the        pixel intensity of said pixel in the RGB training image, wherein        a_(CS-cand) and b_(CS-cand) are one of the possible combinations        of a candidate a-value and a candidate b value, and wherein        DERl_(acand-bcand) is the first derivative value computed for        each pixel in the derivative image by using said one combination        of candidate a- and b-values;    -   for each of said computed derivative images, computing a color        separation metric J_(CS.acand-bacand) according to the formula        J_(CS.acand-bcand)=Σ_(x)Σ_(y)(|DER1_(acand-bacand)(x,y)−DER1_(acand-bacand)(x,y−s)|²),    -   wherein x and y are the x and y coordinates of the pixels in the        derivative image computed for one of the training images and for        one combination of an a-candidate value and a b-candidate value,        and s is a number of pixel distance positions, whereby s has        been empirically determined as pixel distance positions that        correlate with high J_(CS) values in images of known, high color        separation quality;    -   identifying the combination of candidate a- and b-values that        correlate best with the color separation scores assigned to each        of the training images and known to accurately indicate the        color separation quality of said training image; and    -   using the identified candidate a- and candidate b values as the        parameter values a_(CS), b_(CS) having been empirically        determined.

For example, the a-value and the b-value respectively may are varied inthe range [−1,1] in steps of 0.2 for computing the candidate a- andb-values; so each of a and b can take up to 9 possible value and the set(a_(F), b_(F)) can have 9×9=81 possible parameter value pairs for thefocus metric. The set of candidate a_(CS) and b_(CS) candidate valuesmay be computed in the same way, thereby generating a set of 9×9=81possible parameter value pairs for the color separation metric. For the1140 images in the z-stack based training set, the correlation betweenthe proposed features (s pixel differences using R+a*G+b*B) with theempirically determined GT scores is determined as described alreadybefore.

According to embodiments, each of the digital images is an RGB image.The computation of the focus metrics comprises, for each of the digitalimages:

-   -   computing an optical-density-domain image from the digital RGB        image by computing a ROD, a GOD and a BOD value for each pixel        of the RGB image according to:        ROD=−log(R/BK), GOD=−log(G/BK), BOD=−log(B/BK),        wherein R, G and B are the red, green and blue components of the        pixel intensity of said pixel in the RGB image, wherein BK is a        background intensity value typical for tissue slide glass;        [typically a value between 230-250, e.g. 240. The same BK value        may be used for computing ROD, GOD and BOD. Thus, an        optical-density-domain image is an image whose pixels        respectively comprise a ROD, GOD and BOD value computed from the        RGB values of a respective pixel of an original image as        described;    -   computing a derivative image from the optical domain image by        computing a second derivative value DER2 for each of the pixels        in the optical-density-domain image according to        DER2=ROD+a_(F)*GOD+b_(F)*BOD, where −1≤a_(F)≤1, where        −1≤b_(F)≤1, wherein a_(F) and b_(F) are parameters having been        empirically determined as values which, when used for computing        focus metrics, provide focus metrics having predictive power in        respect to whether an image is in-focus;    -   computing a focus metric J_(F) according to the formula        J _(F)=Σ_(x)Σ_(y)(|DER2(x,y)−DER2(x,y−s)|²),

wherein x and y are the x and y coordinates of the pixels in thederivative image and s is a number of pixel distance positions, wherebys has been empirically determined as pixel distance positions thatcorrelate with high J_(F) values in images known to be in-focus. S_(CS)and S_(F) may be identical, e.g. 2 pixel, 3 pixel or 4 pixel.

According to embodiments, the method further comprises empiricallydetermining the parameters a_(F) and b_(F) by:

-   -   obtaining a plurality of different candidate a-values and        candidate b-values for a_(F) and b_(F) , each candidate a-value        and each candidate b-value lying in the range [−1; +1];    -   obtaining a plurality of training z-layers in a z-stack of        training digital images, each training digital image being an        RGB image having assigned a focus score known to accurately        indicate whether the training image is in-focus;    -   computing a training optical-density-domain image for each of        the training images by computing a ROD, a GOD and a BOD value        for each pixel of the training image; the computation of the        ROD, a GOD and a BOD values is performed like already explained        above;    -   for each possible combination of a candidate a-value and a        candidate b-value, and for each of the training        optical-density-domain images in the training z-stack, computing        a derivative image according to the formula        DER2_(acand-bcand)=ROD+a_(F-cand)*GOD+b_(F-cand)*BOD, wherein        ROD, GOD and BOD respectively are components of a pixel in the        training optical-density-domain image, wherein a_(F-cand) and        b_(F-cand) are one of the possible combinations of a candidate        a-value and a candidate b-value and wherein DER2_(acand-bcand)        is the second derivative value computed for each pixel in the        derivative image by using said one combination of candidate a-        and b-values;    -   for each of said computed derivative images, computing a focus        metric J_(F.acand-bacand) according to the formula        J_(F.acand-bcand)=Σ_(x)Σ_(y)(|DER2_(acand-bacand)(x,y)−DER2_(acand-bacand)(x,y−s)|²),    -   wherein x and y are the x and y coordinates of the pixels in the        derivative image computed for one of the training images and for        one combination of an a-candidate value and a b-candidate value,        and s is a number of pixel distance positions, whereby s has        been empirically determined as pixel distance positions that        correlate with high J_(F) values in images of known focus        quality;    -   identifying the combination of candidate a_(F-cand) and        b_(F-cand)—values that correlate best with the focus scores        assigned to each of the training images and known to accurately        indicate the focus quality of said training image; and    -   using the identified candidate a_(F-cand)—and candidate        b_(F-cand) values as the parameter values a_(F), b_(F) having        been empirically determined.

For example, the focus scores assigned to each of the training imagesand known to accurately indicate the focus quality of said trainingimage could be manually annotated scale values indicating a degree of adigital image being “in-focus”. The color separation scores assigned toeach of the training images and known to accurately indicate the colorseparation quality of said training image could likewise be manuallyannotated scale values indicating a degree of a digital image beingcapable of correctly determining if two dots have the same or differentcolors and thus may be indicative of different biomarkers. In sum, aplurality of candidate focus metrics and candidate color separationmetrics may be computed and evaluated automatically and may becorrelated with metrics acting as “training” or “ground true” metrics.

According to embodiments, the method further comprises:

-   -   obtaining a plurality of training z-layers in a z-stack of        training digital images, each training digital image being an        RGB image having assigned a color separation score known to        accurately indicate the color separation quality of the training        image;    -   for each of the training digital images computing a plurality of        candidate color separation metrics, the plurality of candidate        color separation metrics comprising metrics being incapable of        indicating color separation quality and comprising metrics        indicating color separation quality;    -   for each of the candidate color separation metrics, generating a        plot 902 comprising an x-axis and a first and a second y-axis,        the x-axis representing the z-axis of the z-stack of training        images, the first y-axis representing units of the color        separation scores assigned to the training images, the second        y-axis representing units of the candidate color separation        metrics, the plot generation comprising plotting the color        separation quality scores assigned to the training images over        the x-axis and the first y-axis and plotting the computed        candidate color separation metrics computed for the training        images over the x-axis and the second y-axis;    -   selectively using the candidate color separation metrics whose        change across the z-layers correlates with the change of the        color separation scores assigned to the training images across        the z-layers for computing the color separation metrics for the        single tissue sample.

A candidate color separation metrics is indicative of a property of adigital image that is suspected of having predictive power in respect towhether a blob of pixels has a first or a second color. A candidatecolor separation metrics may comprise one or more numerical or otherdata values that are computed by one or more image analysis operationsfrom a digital image. A color separation metrics has alone or incombination with other color separation metrics predictive power inrespect to whether a blob of pixels has a first or a second color.

Analogously, a candidate focus metrics is indicative of a property of adigital image that is suspected of having predictive power in respect towhether an image is in-focus. A candidate focus metrics comprises one ormore numerical or other data values that are computed by one or moreimage analysis operations from a digital image. A focus metrics hasalone or in combintion with other focus features predictive power inrespect to whether an image is in-focus, i.e., the focus quality of animage.

According to embodiments, the method further comprises:

-   -   obtaining a plurality of training z-layers in a z-stack of        training digital images, each training digital image being an        RGB image having assigned a focus score known to accurately        indicate the focus quality of the training image;    -   for each of the training digital images computing a plurality of        candidate focus metrics, the plurality of candidate focus        metrics comprising metrics being incapable of indicating focus        quality and comprising metrics indicating focus quality;    -   for each of the candidate focus metrics, generating a plot        comprising an x-axis and a first and a second y-axis, the x-axis        representing the z-axis of the z-stack of training images, the        first y-axis representing units of the focus scores assigned to        the training images, the second y-axis representing units of the        candidate focus feature, the plot generation comprising plotting        the focus quality scores assigned to the training images over        the x-axis and the first y-axis and plotting the computed        candidate focus metrics computed for the training images over        the x-axis and the second y-axis; and    -   selectively using the candidate focus features whose change        across the z-layers correlates with the change of the focus        scores assigned to the training images across the z-layers for        computing the focus metrics for the single tissue sample. For        example, in case a pair of Gaussian filters with a particular        standard deviation have been identified (by computing a        difference-of-Gaussian image from said pair of filters and        determining that the change in DoG values correlate with the        change in a ground truth metrics obtained from training images)        as parameters having high predictive power in respect to color        separation while another pair of Gaussian filters with another        standard deviation have been identified as parameters having no        predictive power in respect to the color separation, only the        first DoG filter pair will be used for computing the color        separation metrics on the test images (i.e., the z-stack of        digital images depicting the single tissue sample on the slide).        Metrics vectors of parameters having no predictive power in        respect to the color separation will show a change in value from        z-layer to the next (the “vector slope”) which does not or not        very well correlate with the change in value of a vector of        ground truth score or color separation metrics obtained from        training images of other tissue samples having been treated        basically in the same manner as the single tissue sample used        for generating the z-stack of test images.

According to embodiments, the determining of the one of the z-layers tobe used for computing tissue-related or cell-related scores is performedwhile a process of scanning a slide comprising the tissues sample forgenerating the z-stack of images of said tissue sample is in progress.Thus, the z-layer identification may be performed in real time, becausethe metrics not being based on dot-specific features can be computedvery fast. For example, the identified z-layer can be the layer that isdisplayed as a control image during the scanning process, i.e. duringthe process of a scan apparatus moving over the slide and generating theimage pixel-line by pixel-line.

According to embodiments, the method further comprises:

-   -   identifying a plurality of tiles in the z-stack of digital        images of the single tissue sample, each tile comprising, for        each of the z-layers, a sub region of the image;    -   for each of the tiles and for each of the z-layers, computing a        plurality of tile focus metrics, each tile focus metric being an        indicator of focus quality of the sub region of the digital        image of said z-layer within said tile and being derived from        image data of said sub region;    -   for each of the tiles and for each of the z-layers, computing a        plurality of tile color separation metrics, each tile color        separation metric being an indicator of color separation quality        of the sub region of the digital image of said z-layer within        said tile and being derived from image data of said sub region;    -   evaluating the tile focus metrics and the tile color separation        metrics of all the tiles and selecting a sub-set of the tiles in        dependence on the tile focus metrics and tile color separation        metrics of the tiles; and    -   selectively using the sub-set of the tiles for additional        automated image analysis operations and/or highlighting the        sub-set of the tiles in a graphical user interface.

This may be advantageous as a fine-grained tile structure andtile-specific focus and color separation scores may allow to selectivelydisplay and/or further process the ones of the tiles having high colorseparation scores and high focus scores.

A tile focus metrics is also referred herein as focus feature or focusmetric computed for a tile from image data covered by said tile. A tilecolor separation metrics is also referred herein as color separationfeature or color separation metrics computed for a tile from image datacovered by said tile. A tile as used herein is a sub region of a digitalimage. A tile may have any shape, e.g. a circle, a square or arectangle. The tiles of an image may be disjoint or overlapping.

In a further aspect the disclosure relates to an image analysis methodfor selecting tiles in a multi-layered, z-stack of digital images. Thez-stack of digital images depict different layers of a single tissuesample located on a slide. The z-stack images have been acquired byvarying the focus of an optical sensor, e.g. the sensor of a slidescanning system. The method comprises:

-   -   identifying a plurality of tiles in the z-stack of digital        images of the single tissue sample, each tile comprising, for        each of the z-layers, a sub region of the image;    -   for each of the tiles and for each of the z-layers, computing a        plurality of tile focus metrics, each tile focus metric being an        indicator of focus quality of the sub region of the digital        image of said z-layer within said tile and being derived from        image data of said sub region;    -   for each of the tiles and for each of the z-layers, computing a        plurality of tile color separation metrics, each tile color        separation metric being an indicator of color separation quality        of the sub region of the digital image of said z-layer within        said tile and being derived from image data of said sub region;    -   evaluating the tile focus metrics and the tile color separation        metrics of all the tiles and selecting a sub-set of the tiles in        dependence on the tile focus metrics and tile color separation        metrics of the tiles; and    -   selectively using the sub-set of the tiles for additional        automated image analysis operations and/or highlighting the        sub-set of the tiles in a graphical user interface.

For example, the z-stack of digital images may be an already scannedimage comprising multiple image layers obtained for different settingsof the optical focus. The scanned image may e.g. be a 40000×40000 scan,at 40× resolution corresponding to pixels that are approximately 0.25microns by 0.25 microns. In the following, embodiments of the inventionwill be described which may be likewise be performed on z-stacks ofdigital images which in which tiles may have been identified and onz-stacks of digital images where no tiles were identified. Only featuresexplicitly relying on the presence of tiles are limited to embodimentscomprising a tile identification.

According to embodiments, the tiles are disjoint squares, e.g. 320×240pixel rectangles. In some embodiments, the time spent in computing thefocus metric for such a tile is in the order of a few msec. This tilesize is advantageous as it ensures that tile score computation for a40000×40000 pixel scan, at 40× resolution, can be done in within 12-15min on e.g. the Coreo scanner.

According to embodiments, the method further comprising automaticallyidentifying a plurality of first dots and a plurality of second dots ineach of the images of the z-stack of images, each dot being a blob ofadjacent pixels, the pixel intensity values of the first dotscorrelating with an optical signal intensity of a first stain, the pixelintensity values of the second dots correlating with an optical signalintensity of a second stain, the first and second stain having been usedfor staining the single tissue sample and having different colors. A“dot” as used herein, that may also be considered as a small blob, is aset of adjacent pixels with similar optical features, e.g. whoseintensity values lie above a threshold. A dot may be identified by aconnected component analysis operation, e.g. intensity thresholding ofan RGB image or the like.

According to embodiments, the method further comprises computing one ormore of the tile focus metrics by:

-   -   applying a plurality of pairs of Gaussian filters on each of the        digital images of the z-layer of images, the kernels of the        Gaussian filters having different standard deviations in each        pair, for computing, for each of the z-layer images and for each        of the pairs of Gaussian filters, a difference-of-Gaussian        image,    -   identifying, in each of the difference-of-Gaussian images, a        plurality of DoG dots, a DoG dot being a set of adjacent pixels        in a difference-of-Gaussian image whose pixel value exceeds a        DoG threshold;    -   computing, for each of the tiles and for all sub regions of        z-stack images covered by said tile, an average-DoG-dot-value        for all DoG dots identified in the difference-of-Gaussian image        derived from said image sub-region, the average-DoG-dot value        being indicative of the average DoG value of all pixels in the        difference-of-Gaussian image contained in said DoG dot;    -   computing, for each of the tiles and for all sub regions of        z-stack images covered by said tile, the mean of all        average-DoG-dot-values computed for said tile and using said        mean of all average-DoG-dot-values as one of the tile focus        metrics; and/or    -   computing, for each of the tiles and for all sub regions of        z-stack images covered by said tile, the median of all        average-DoG-dot-values computed for said tile and using said        median of all average-DoG-dot-values as one of the tile focus        metrics; and/or    -   sorting the average-DoG-dot-values computed for each of the        tiles and for each of the z-layers of said tile and computing,        for each of the tiles and for all sub regions of z-stack images        covered by said tile, the mean of a predefined percentile value        of the highest ranking ones of the sorted average-DoG-dot-values        computed for said tile and using said mean of all        average-DoG-dot-values as one of the tile focus metrics. For        example, the percentile value may be 50%.

The pair of Gaussian filters is applied in order to perform a featureenhancement step. It involves the subtraction of one blurred version ofan original image from another, less blurred version of the original.The blurred images are obtained by convolving an original image with twoGaussian filters whose kernels have differing standard deviations.Blurring an image using a Gaussian kernel suppresses only high-frequencyspatial information. Subtracting one image from the other preservesspatial information that lies between the range of frequencies that arepreserved in the two blurred images. Thus, the difference of Gaussiansis a band-pass filter that discards all but a handful of spatialfrequencies that are present in the original grayscale image.Preferentially, the standard deviation of the DoG filter is chosen suchthat dots having a size of 2-6 pixels and clusters of such dots willgenerate peaks in a DoG image.

For example, a first pair of Gaussian filters may have a first filterwith a Kernel of one standard deviation and a second filter with aKernel of two standard deviations. A second pair of Gaussian filters mayhave a first filter with a Kernel of two standard deviations and asecond filter with a Kernel of three standard deviations. A third pairof Gaussian filters may have a first filter with a Kernel of threestandard deviations and a second filter with a Kernel of four standarddeviations. By applying all said pairs of Gaussian filters on eachdigital image of the z-stack, it is possible to identify the particularones of the Gaussian filter pairs yielding particularly high DoG valuesin a difference-of-Gaussian image at image areas where dots are located.Preferentially, the size of the Kernels are adapted to the expectedaverage size of the dots, e.g. are between 2 and 15 pixels in diameter.

According to embodiments, the method further comprising computing one ormore of the tile focus metrics by:

-   -   identifying, in each of the difference-of-Gaussian images, a        plurality of first DoG dots, a first DoG dot being a set of        adjacent pixels in a difference-of-Gaussian image whose pixel        value exceeds a DoG threshold and which overlays one of the        identified first dots;    -   identifying, in each of the difference-of-Gaussian images, a        plurality of second DoG dots, a second DoG dot being a set of        adjacent pixels in a difference-of-Gaussian image whose pixel        value exceeds a DoG threshold and which overlays one of the        identified second dots;    -   computing, for each of the tiles and for all sub regions of        z-stack images covered by said tile, an        average-first-DoG-dot-value for all first DoG dots identified in        the difference-of-Gaussian image derived from said image        sub-region, the average-first-DoG-dot value of a first DoG dot        being indicative of the average DoG value of all pixels in the        difference-of-Gaussian image contained in said first DoG dot;    -   computing, for each of the tiles and for all sub regions of        z-stack images covered by said tile, an        average-second-DoG-dot-value for all second DoG dots identified        in the difference-of-Gaussian image derived from said image        sub-region, the average-second-DoG-dot value of a second DoG dot        being indicative of the average DoG value of all pixels in the        difference-of-Gaussian image contained in said second DoG dot;    -   computing, for each of the tiles and for all sub regions of        z-stack images covered by said tile, the mean of all        average-first-DoG-dot-values computed for said tile and using        said mean of all average-first-DoG-dot-values as one of the tile        focus metrics; and/or    -   computing, for each of the tiles and for all sub regions of        z-stack images covered by said tile, the median of all        average-second-DoG-dot-values computed for said tile and using        said median of all average-second-DoG-dot-values as one of the        tile focus metrics.

According to embodiments, the method further comprises computing one ormore of the tile color separation metrics by:

-   -   representing each of the identified first and second dots as a        dot-vector in a color space;    -   finding the spherical coordinates of the dot-vector of each of        the first and second dots for identifying, for each of the first        and second dots, an azimuth angle 972 and an elevation angle 974        of the dot-vector represented in spherical coordinates;    -   for each of the tiles and for each z-layer of said tiles,        plotting the azimuth angles and elevation angles of all        identified first and second dots contained in an        azimuth-elevation plot 990;    -   fitting a first ellipse around all data points in the        azimuth-elevation plot 990 having been derived from one of the        first dots;    -   fitting a second ellipse around all data points in the        azimuth-elevation plot 990 having been derived from one of the        second dots;    -   computing the distance of the first and second ellipse; and    -   using the computed distance as one of the tile color separation        metrics, whereby the larger the distance between the first and        second ellipse, the higher the color separation quality.

A corresponding example is depicted and explained for example in thedescription of FIG. 8.

According to embodiments, representing each of the identified first andsecond dots as a dot-vector in a color space comprises:

-   -   for each of the identified first and second dots, computing an        R_(DOTAVG) value, a G_(DOTAVG)-value and a B_(DOTAVG) value, the        R_(DOTAVG) value being the average of the R component of all        pixels contained in said dot, the G_(DOTAVG) value being the        average of the G component of all pixels contained in said dot,        the B_(DOTAVG) value being the average of the B component of all        pixels contained in said dot, the RGB values being red green and        blue pixel values in RGB color space;    -   for each of the identified first and second dots, computing an        optical density vector RGBOD (920-934) comprising the elements        ROD_(DOTAVG), GOD_(DOTAVG) and BOD_(DOTAVG) computed for the dot        according to:        ROD_(DOTAVG)=−log(R _(DOTAVG) /BK), GOD_(DOTAVG)=−log(G_(DOTAVG)        /BK), BOD_(DOTAVG)=−log(B _(DOTAVG) /BK),    -   Using the optical density vectors RGBOD as the dot-vector of the        first and second dots.

According to embodiments, the tile color separation metrics comprisesone or more of:

-   -   the maximum value (Amax) observed in all pixels of a dot whose        RGB values are represented in a color channel, the color channel        being a color channel in the LAB-color space whose color is the        most similar to the color of the first or the second stain;    -   the maximum value (Asigmax) observed in all pixels of a dot        whose RGB values are represented in a color channel derivative,        the color channel derivative being computed as a derivative of a        color channel in the LAB-color space whose color is the most        similar to the color of the first or the second stain, the        computation of the derivative color channel comprising applying        a sigmoid function for removing extreme values;

For example, the color channel derivative (“Asig image”) may be aderivative version of a LAB color channel, e.g. the A-channel of thedigital image, generated by applying a sigmoid function on pixel valuesof the A-channel image . The Asig image is obtained by applying asigmoidal function on said LAB-color channel image. Thereby, theA-channel intensity values are normalized to a value between 0 and 1. Inother words, by applying the sigmoidal function, the LAB-color-channelpixel values are nonlinearly transformed by the sigmoidal function,whereby the sigmoidal function limits the range of the output(“normalized” or “smoothed”) intensity values to values between 0 and 1.The sigmoid function is, for example, almost linear near the meancolor-channel pixel intensity value and has smooth nonlinearity at bothextremes, ensuring that all (“Asig”) pixel values output by the sigmoidfunction are within a limited range, whereby the influence of extremevalues or outliers in the data is reduced. The idea is that there issome linear range where increases or decreases in the metric value canbe considered “realistically” linear in the real world. At extremevalues, a color channel pixel intensity may start to becomeasymptotically more meaningless. One might say that, for example, avalue of 253 is really no different than a value of 254 and as such eachwould get transformed by the sigmoid function to some value near 1. Samecould be said about color-channel intensity values in the range of 1-3units which may be transformed by the sigmoid function to a valueasymptotically close to 0.0. So the Asigmax of a dot is the maximumvalue (Asigmax) observed in all pixels of a dot in the Asig image.

-   -   the maximum color channel value observed in all pixels of a dot        whose RGB values are represented in said color channel, the        color channel having been generated by a color deconvolution of        the RGB values into at least the first color of the first stain        and the second color of the second stain, the color channel        generated by the color deconvolution corresponding to the first        or the second color;    -   a radial symmetry value indicating radial symmetry of the dot;    -   an average DoG value computed from the DoG values of all pixels        of a dot, the DoG values of the dot being obtained from a        difference-of-Gaussian image area in a difference-of-Gaussian        image corresponding to the location of the pixels of said dot.

The radial symmetry value can be computed for example from votes of agradient image derived from at least one image channel. Said imagechannel may be for example a luminance channel, an unmixed red channel,an A channel, an Asig channel, or a green channel; It is assumed thatsome dots generated by an optical signal of a first stain selectivelybound to a first biomarker may be significantly more radial symmetric(on average) than dots generated by an optical signal of a second stainbound to another biomarker; thus, radial symmetry may be a metricshaving predictive power in respect to the color of a dot; also the abovementioned color separation metrics are metrics having been observed tohave predictive power in respect to the color (and thus, typically, to aparticular biologic feature such as the presence or relative amounts ofbiomarkers) and thus can be used as color separation metrics.

According to embodiments, the method further comprising generating aheat map for the tiles, the heat map being indicative of the focusquality and/or of the color separation quality computed for each of thetiles, the highlighting of the sub-set of the tiles in a graphical userinterface comprising displaying the computed heat map as an overlay ofthe tiles. A heat map is a graphical representation of data where theindividual values contained are represented as colored pixels or coloredregions, or in the case of monochrome, as pixels or regions of varyingintensity. For example, the color of the heat map may indicate the focusscore and/or the color separation score of an image.

According to embodiments, each tile has a width and height of 250-350pixels, wherein preferentially 1 pixel corresponds to 0.23 μm of thetissue slide. Said size has been observed to be a good compromise ofavoiding using too small tiles what could increase CPU consumption andof avoiding overly large tiles resulting in a loss of precision of thevalidity of tile-specific scores.

According to embodiments, a focus quality scores and a color separationquality score are computed only for those tiles having at least onefirst dot, each first dot corresponding to an in situ hybridizationsignal having a first color and at least one second dot, each second dotcorresponding to an in situ hybridization signal having a second color.This may save CPU resources.

In a further aspect the disclosure relates to an image analysis systemcomprising one or more processors and a storage medium comprisingcomputer-interpretable instructions which, when executed by the one ormore processors, cause the one or more processors to perform a methodaccording to any one of the embodiments of the invention describedherein. The image analysis system can be, for example, a digital dataprocessing device, e.g. a computer, comprising an interface forreceiving image data from a slide scanner, a camera, a network and/or astorage medium.

In a further aspect the invention relates to a system comprising theimage analysis system according to embodiments of the invention andcomprising an optical slide scanning system. The slide-scanning systemis configured for generating the z-stack of the digital images of thesingle biological sample. The instructions, when executed by the one ormore processors, cause the one or more processors to perform the methodaccording to embodiments of the invention for selecting the one of thez-layer image in real time for already generated parts of the z-stack ofthe digital images while the slide-scanning system scans the tissueslide.

In a further aspect the invention relates to a non-volatile storagemedium comprising instructions which, when executed by a processor,cause the processor to perform the method of any one of the embodimentsdescribed herein.

EXAMPLE 1

Dual ISH HER2

In general, to automatically score a breast tissue sample, candidatenuclei are selected for quantitative analysis. The computer system 14automatically counts different features (e.g., HER2 genes, chromosome17s, etc.) and determines the ratio of the number of features. Adiagnosis can be made based, at least in part, on the ratios. Toevaluate whether the tissue sample (e.g., breast tissue) is a carcinoma,the computer system 14 can assist the user in obtaining informationabout the selected region by, for example, detecting the amplificationof genes by evaluating the ratio of the number of HER2 gene signals tothe number of chromosome 17 signals.

When using the Ventana INFORM HER2 dual ISH assay, to determine aHER2/chromosome 17 ratio, the imaging apparatus 12 captures images thatinclude silver in situ hybridization signals, red in situ hybridizationsignals, or the like. More, specifically, and with reference to FIG. 6,the HER2 gene is detected by a dinitrophenyl (DNP) labeled probe (A) andvisualized utilizing VENTANA ultraView Silver ISH DNP (SISH) Detection(B). The chromosome 17 centromere is targeted with a digoxigenin (DIG)labeled probe (A) and detected using VENTANA ultraView Red ISH DIGDetection (C). Dual ISH staining utilizing this dual detection resultsin visualization via light microscopy in which HER2 appears as discreteblack signals (SISH) and Chr17 as red signals (Red ISH).

The tissue is scored based on the signals corresponding to HER2 genesand chromosome 17s to determine the HER2/chromosome 17 ratio. Based onthe ratio, the specimen's HER2 gene is determined to be amplified or notamplified. If the determined score is Score<=1.8=>Bin 1 (regarded asnon-amplified or negative, from a score perspective); if the Score>1.8and Score<=2.2=>Bin 2 (when the score is in this range, it is difficultto make a decision and 20 more cells are considered to decideamplified/non-amplified); and if the Score>2.2=>Bin 3 (regarded asamplified or positive).

EXAMPLE 2

Description of Experimental Dataset and Correlation Metrics

For training the algorithm, i.e. to compute the correlation between theground truth and computed features and to find the features morecorrelated to the ground truth, volume scans of 10 Dual ISH slides wereused that were scanned at 40× using 15 z-layers at 0.25 micron spacingon the Coreo scanner. 76 Fields of View (FOVs) were extracted, resultingin 76*15=1140 images. For testing how well the correlation holds forunseen data, the testing set included volume scans of 12 Dual ISHslides, scanned at 40× using 15 z-layers and 0.25 micron separation;where 45 FOVs were extracted resulting in 45×15=675 images.

Suppose (a,b) (where derived image being considered=R+a*G+b*B, where(R,G,B) can be original 8-bit pixel values, or they can refer to thecorresponding optical density values) are varied in the range [−1,1] insteps of 0.2; so each of a and b can take up 9 possible value and theset (a, b) can have 9×9=81 possible values. For the 1140 images in thez-stack based training set, the correlation between the proposedfeatures (2nd pixel differences using R+a*G+b*B) with the GT scores wasstudied.

Each image was assigned a ground truth focus score and a ground truthred-vs-black separation score. The assigned focus score was in the range[0-2] where 2 indicated “very good focus” while a low score in the range[0-1] indicated very poor focus. For the red-vs-black separation, ascore of [1-2] indicated poor red-vs-black discriminability, a score of[4-5] indicated much better red-vs-black discriminability, while a scoreof 3 indicated borderline red-vs-black discriminability.

Given a z-stack, ground truth focus and red-black scores were assignedto all the individual layers. If a feature was indicative of focus, thenas the focus changed per layer, the feature should haveincreased/decreased as the focus term increased/decreased. Thus, theexpectation was that the proposed focus feature would have been able totrack the change in focus score across the different z-layers. Afunction of +1 was used when the feature and ground truth score (thefeature and score can be in the context of focus, or in the context ofred-vs-black separation) both increased or both decreased or are bothvery close to zero; −1 when the signs were different; and 0.5 when oneof the feature changes was close to zero.

EXAMPLE 3

Computation of Correlation score (C) Between a Quality Score FeatureVector (F), and a Ground Truth Score Vector(G). The following steps maybe performed for a plurality of quality score feature vectors, whereby aquality score feature vector may be a vector of candidate colorseparation metrics computed for each of the z-layer images or may be avector of candidate focus metrics computed for each of the z-layerimages. For example, the correlation may be computed by using a plot 902as depicted in FIG. 9 for a candidate focus feature metrics.

A plurality of training z-layers in a z-stack of training digital imagesis read by a processor of the computer system. Each training digitalimage is an RGB image having assigned a focus score known to accuratelyindicate the focus quality of the training image. For each of thetraining digital images, the processor computes a plurality of candidatefocus metrics. It is at this stage not yet clear if a candidate focusmetric has any predictive power in respect to focus quality as theplurality of candidate focus metrics may comprise a mixture of metricsbeing incapable of indicating focus quality and metrics capable ofindicating focus quality. For each of the candidate focus metrics, theprocessor generates a plot 902 comprising an x-axis and a first and asecond y-axis, the x-axis representing the z-axis of the z-stack oftraining images, the first y-axis representing units of the focus scoresassigned to the training images, the second y-axis representing units ofthe candidate focus metrics. In the case of the candidate colorseparation metrics (not shown) the first y-axis would relate to a groundtruth value of the color separation metrics used as the candidate colorseparation metrics and the second y-axis would represent the candidatecolor separation metrics. The plot generation comprises plotting theground truth score vector G (in the example of FIG. 9: the focus qualityscores 904 assigned to the training images) over the x-axis and thefirst y-axis and plotting the quality score feature vector F (in theexample of FIG. 9: the candidate focus metrics 906 computed for thetraining images) over the x-axis and the second y-axis.

Then, the candidate focus metrics whose change across the z-layerscorrelate with the change of the focus scores assigned to the trainingimages across the z-layers are selectively used, in a later applicationof the method on the z-stack of digital images (which may act as “testimage”, not as a “training image”) for computing the focus metrics.

It was assumed that both F and G were vectors with N elements. N mayindicate the number of layers in the z-stack of images, e.g. 15. Anormalized difference vector DF is computed from vector F and anormalized difference vector DG is computed from vector G. The 1-Dnormalized difference was computed for both vectors DF and DG, both with(N−1) elements; where, DF(n)={F(n+1)−F(n)}/{0.0001+F(n)};DG(n)={G(n+1)−G(n)}/{0.0001+G(n)}; 1<=n<N. These normalized differenceterms reflected the slope of the vectors, and when the slope term wasless than α (in our experiments, a was set to 0.0005) in magnitude, itwas regarded as 0.

The correlation score C was computed as:

C={Σi f(DF(i), DG(i))}/(N−1), 1<=i<N; where the function f(.,.) isdefined such that:

f(x,y)=+1, when x and y are both >α, or both <−α, or both magnitudeswere within |α|

f(x,y)=−1, when one of them (x,y) exceeds α, and the other was less than−α

f(x,y)=0.5, when one of them (x,y) was within |α| and the other exceeds|α| and was of a different sign. The parameter |α| is a minimum slopechange threshold that may be used for identifying a very low change inslope of at least one of the normalized vectors DG or DF from onez-layer to another.

Thus, when the slopes of the 2 vectors F and G were similar, thecorrelation score between the 1-D normalized difference vectors, DF andDG, was close to +1. In this case, a respective metrics may beconsidered as having good predictive power in respect to focus or colorseparation.

EXAMPLE 4

FIG. 10a depicts the representation of 6 dots of a first color and 6dots of a second color in an optical density domain coordinate system.For example, a set of 6 vectors 918 corresponds to a set of 6 dotsgenerated by black color signals of respective tissue sample areasstained with a black stain such as silver. A set of 6 vectors 936corresponds to a set of 6 dots generated by red color signals ofrespective tissue sample areas stained with a red stain, e.g. a redfluorescent stain.

At first, the optical density (OD) of the rgb values for the 12identified dots are determined. For example, the 12 dots may all becontained in the same tile.

Each of the identified first dots (e.g. black dots) and second dots(e.g. red dots) are represented as dot-vectors. The number 918 refers tothe set of dot vectors derived from black dots and 936 refers to the setof dot vectors derived from red dots.

The 12 dot vectors are represented in spherical coordinates and theazimuth and elevation are identified.

FIGS. 10b and 10c depict a projection of the 12 dot vectors into theunit sphere 970, 980 for illustrating the identification of the azimuthangle 972 and the elevation angle 974 for each of the first and seconddots. The azimuth angle 972 and the elevation angle 974 of each dot arederived from the spherical coordinates of a corresponding dot vector.For each of the tiles and for each z-layer of said tiles, the azimuthangles and elevation angles of all identified first and second dots areplotted in an azimuth-elevation plot 990 as shown in FIG. 11. The datapoints 918′ in the plot 990 correspond to the dot vectors of the blackdots and the data points 936′ in the plot 990 correspond to the dotvectors of the red dots. Then, optionally, a data point classificationstep may be performed for classifying the data points in the plot 990into a first class of data points derived from dots having a first color(e.g. black) and into a second class of data points derived from dotshaving a second color (e.g. black). It may also be the case that thefirst and second classes of dots have been classified previously or thatthe subsequent ellipse fitting operation is part of a classificationprocedure that groups data points in the azimuth-elevation plot intodifferent classes based on the respective azimuth and elevation angles.

The processor fits a first ellipse around all data points in theazimuth-elevation plot 990 having been derived from one of the firstdots and fits a second ellipse around all data points in theazimuth-elevation plot 990 having been derived from one of the seconddots as described also in more detail for FIG. 7 a.

Ellipse Fitting to Red and Black Dots

Ellipses were fitted to the RGB optical density (OD) values of thepixels identified as red or black, after dots were found. Ellipsefitting is visually shown, for example, in FIG. 7A.

The RGB-OD vector:

RGBOD(i)=log(BK/RGB(i)), where background color BK=240, i=1,2,3 (idenoted the i th color channel).

The RGB-OD vector was projected to the unit sphere and the abscissa wasazimuthal angle (φ) and the ordinate was elevation angle (θ). Therelation between Cartesian (x, y, z) and spherical (r, θ, φ) coordinatesis as follows: x=r.sin(θ).cos(φ); y=r.sin(θ).sin(φ); z=r.cos(θ).

Best features for red-black quality were “yMu(2)” and “dMu,” “where“yMu(2)” is the mean elevation of the OD vector for the red pixels and“dMu” is the Euclidean distance between the center of the red pixels andthe center of the black pixels in the elevation-azimuth plot. It wasdetermined that the farther apart the red and black pixel clusters were,the larger dMu became. This was demonstrated in FIGS. 7A and 7B, whereFIG. 7B show the cell image where the red and black dot pixels have beenidentified, and based on them, the ellipse plot for the red and blackdots have been computed, as shown in FIG. 7A. Two representative pixelswere: Black RGB=108, 114, 150; Red RGB=177, 123, 178; RGBOD, black:0.84, 0.78, 0.51; RGBOD, red: 0.35, 0.71, 0.34; Elev-azim, black: 0.74,0.42; Elev-azim, red: 1.04, 0.41.

EXAMPLE 5

Superior Results of New Focus Method Compared to the Prior Art

227 z-stacks were obtained using 15 layers per tile, and using 0.25micron spacing, performed on the Coreo scanner. A range of layers were“ground truthed” and marked corresponding to “very good qualityscoreable layers” for the given z-stack. The expectation from a scoringmetric was that when a best layer was computed algorithmically, thenthat best layer should belong to the set of “very good quality scoreablelayers,” which were identified visually. The best layer was determinedusing the old focus method (using sum of squared difference based costcomputed on Green channel) and also using the new method and the resultswere compared.

Based on the data collected from 227 z-stacks, it was observed that thebest z-layer (as indicated from ground truthing) was captured accuratelyin 187 cases using the new focus method and in only 140 cases using theold focus method. Thus, new focus metrics have been identified, whichare fast to compute, and which return the best z-layer in asignificantly higher number of cases as compared to the old focusmethod. Therefore, the new method provides superior and unexpectedresults as compared with that known in the art.

EXAMPLE 6

(a) Features derived from mean and standard deviation for ellipsesfitted to differently colored dots (e.g. black and red colored dots)

Features: xmu1, xmu2, ymu1, ymu2, xsig1, xsig2, ysig1, ysig2.Considering ellipses plotted for black dot pixels, the center obtained(xmu1, ymu1) where the 2D axis used was x axis=azimuthal coordinate fromOD space and y axis=elevation angle obtained from OD space. Similarly,from red dot pixels, the center computed=(xmu2, ymu2).

The standard deviation along the x-axis (azimuthal coordinate) for blackpixels and red pixels was given by xsig1 and xsig2, respectively. Thestandard deviation along the y-axis (elevation angle) for black pixelsand red pixels was given by ysig1 and ysig2, respectively.

(b) Features derived from the distance between differently coloredellipses (e.g. distance between black and red color ellipses, i.e.,distance between an ellipse fitted to azimuth-elevation plot data pointsderived from black dots and an ellipse fitted to azimuth-elevation plotdata points derived from red dots): dmu, dmunorm

dmu=sqrt(dxmu^2+dymu^2);

dxmu=(xmu1−xmu2); dymu=(ymu1−ymu2);

dmunorm=sqrt(termx1*termx1+termx2*termx2)+(termy1*termy1+termy2*termy2))

where:

termx1=(xmu1−(xmu1+xmu2)/2)/(0.0001+xsig1)

termx2=(xmu2−(xmu1+xmu2)/2)/(0.0001+xsig2)

termy1=(ymu1−(ymu1+ymu2)/2)/(0.0001+ysig1)

termy2=(ymu2−(ymu1+ymu2)/2)/(0.0001+ysig2)

Thus, dMu was the Euclidean distance between the center of the redpixels and the center of the black pixels in the elevation-azimuthalplot. So, the farther apart the red and black pixel clusters were, thelarger dMu became. The 0.0001 term is incorporated in the division toavoid division by zero. Alternatively, any other value larger than zeroand preferentially smaller than 0.001 could be used.

(c) Features derived from the dimensions of color ellipses: semiaxes ofmajor axis 1, minor axis 1, major axis 2, and minor axis 2

Considering the black pixels, an ellipse was obtained and its dimensionswere given by (major axis 1, minor axis 1). Considering the red pixels,an ellipse was obtained and its dimensions were given by (major axis 2,minor axis 2).

Each of said features may be used for computing a color separationmetrics for a particular tile.

EXAMPLE 7

Method to Select “Top N Good Quality” Regions Given Quality Scores forEvery Tile

This example assumed that there were M tiles in a whole slide scan. Italso assumed that the focus score and color separated score werecomputed and that the differently colored pixels were counted. Thisexample considered red and black dot pixels and a color separationbetween the red and black dot pixels.

For the i-th tile, the focus score was F(i), the red-black-separationscore was RB(i), and number of black and red dot pixels were B(i) andR(i), respectively.

A reference value of the focus score was FC (computed empirically basedon the data where we have computed focus scores for a large number ofgood-quality and poor-quality tiles) and a reference value of the redversus black separation score was RBC (computed empirically based on thedata where we have computed red-vs-black separation scores for a largenumber of good-quality and poor-quality tiles).

For the i-th tile, normalized focus score NF(i)=F(i)/FC; For the i-thtile, normalized red-vs-black separation score NRB(i)=RB(i)/RBC. For thei-th tile, the minimum of the normalized focus and normalizedred-vs-black separation scores MlNscore(i)=min(NF(i), NRB(i)).

The minimum number of black dot pixels expected per tile was set asmin_blacks, and the minimum number of red dot pixels expected per tilewas set as min_reds.

Only those tiles considered as valid tiles were those which had acertain minimum number of red tiles and black tiles.

The set of valid tiles, V, out of all M tiles, was:

V={i: B(i)>min_blacks and R(i)>min_reds; 1<=i<=M

The black-to-red ratio (the number of black dot pixels divided by thenumber of red dot pixels) per tile was computed for all the valid tiles.For the k-th valid tile, the ratio term was RATIO(k)=B(V(k))/R(V(k)).

Ratio_sort was set as the sorted version of RATIO, which was sorted indescending order.

Ratio_sorted_cutoff was set as Ratio_sort(round(0.4*|Ratio_sort|));where |Ratio_sort| denoted the number of elements in the set Ratio_sort.

The top 40% of these tiles were taken and marked as valid tiles (Vnew).

Vnew={V(k): Ratio_sort(k)>ratio_sorted_cutoff, 1<=k<=|Ratio_sort|}

The tiles in this set Vnew were then sorted based on MINscore.

MINscore_sorted=sort({MINscore(Vnew(k)); 1<=k<=|Vnew|}), in descendingorder.

The corresponding indices in sorted order was set as indices_sorted.

Therefore, the 1st tile index after sorting MINscore(Vnew) in descendingorder=Vnew(indices_sorted(1)).

To return the top 10% of tiles in Vnew (or 50 tiles, whichever ishigher), then let N=max(round(0.1*|Vnew|), 50).

Then the set of tiles returned ST is:

ST(k)=Vnew(indices_sorted(k)); 1<=k<=N

Thus, it was ensured that the top N regions were selected, which had atleast a minimum of black dot pixels and a minimum of red dot pixels; andthese regions were also high in focus score, and also in red-vs-blackseparation score. By selecting those tiles which had higher value ofMINscore, it was ensured that those tiles were selected which were notvery low with regard to focus, or very low with regard to red-vs-blackseparation. The intuition here was that visually, a tile with moderatequality with regard to focus and red-vs-black separation was believed tohave been preferable over a tile which was of “high focus but lowred-vs-black separation,” or of “high red-vs-black separation but ofvery poor focus.”

Additional Embodiments

1. An image analysis method for determining one of a plurality ofz-layers in a z-stack of digital images to be used for computingtissue-related or cell-related scores, the z-stack of digital imagesdepicting different layers of a single tissue sample located on a slide,the z-stack images having been acquired by varying the focus of anoptical sensor, the method being implemented in an image analysis systemand comprising: for each of the z-layers, computing focus metrics, eachfocus metric being an indicator of focus quality of the z-layer andbeing derived from image data of said z-layer; for each of the z-layers,computing color separation metrics, each color separation metric beingan indicator of color separation quality of the z-layer and beingderived from image data of said z-layer; and evaluating the focusmetrics and the color separation metrics and determining the one of thez-layers to be used for computing tissue-related or cell-related scoresin dependence on the focus metrics and color separation metrics of eachof the z-layers.

2. The image analysis method of embodiment 1, further comprising:analyzing selectively the one of the images in the z-stack of digitalimages that corresponds to the determined z-layer and selectivelycomputing the tissue-related or cell-related scores for said analyzedimage.

3. The image analysis method of any one of the previous embodiments,wherein the evaluation of the focus metrics and the color separationmetrics comprises: computing the difference between the z-layer havingbest focus and the z-layer having best color separation by computing anabsolute value of the difference of the index position of the z-layerhaving best focus and the index position of the z-layer having bestcolor separation in the z-stack; determining whether the computeddifference is greater than, equal to, or less than a predeterminedthreshold value; in case the computed difference is determined to beless than or equal to the predetermined threshold value: computing anaverage of the z-layer index of the z-layer having best focus and thez-layer index of the z-layer having best color separation; if theaverage of the z-layer index is an integer, identifying the one of thez-layers of the z-stack whose index position is identical to thecomputed average of the z-layer indices; if the average of the z-layerindex is not an integer, identifying the two consecutive z-layers of thez-stack which lie next to the computed average of the z-layer indicesand identifying the one of the two consecutive z-layers having thehighest color separation metrics; and using said one identified z-layeras the one z-layer to be used for computing the scores; and in case thecomputed difference is determined to be greater than the predeterminedthreshold value: comparing a focus score derived from the focus metricsof the z-layer image having best focus with a reference focus score, thereference focus score being an empirically determined focus scoreobtained for a digital image depicting the same type of tissue sampleand having been manually identified as being in-focus for determining adeviation of the focus score from the reference focus score; comparing acolor separation score derived from the color separation metrics of thez-layer image having best color separation with a reference colorseparation score, the reference color separation score being anempirically determined color separation score obtained for a digitalimage depicting the same type of tissue sample and having been manuallyidentified as being a high quality color separation image fordetermining a deviation of the color separation score from the referencecolor separation score; in case the deviation of the focus score fromthe reference focus score is greater than the deviation of the colorseparation score from the reference color separation score, using thez-layer image having best color separation for computing the scores.

4. The image analysis method of embodiment 3, further comprising: incase the deviation of the color separation score and the reference colorseparation score is greater than the deviation of the focus score andthe reference focus score, using the z-layer image having best focus forcomputing the scores.

5. The image analysis method of any one of the previous embodiments,further comprising: computing, for each of the z-layers, a focus scoreby using the focus metrics of the digital image corresponding to thez-layer as input; computing, for each of the z-layers, a colorseparation score by using the color separation metrics of the digitalimage corresponding to the z-layer as input; identifying the z-layerhaving best focus and identifying the z-layer having best colorseparation.

6. The image analysis method of embodiment 5, the identification of thez-layer having best focus comprising: storing the focus score of each ofthe z-layers in a respective element of a focus score vector; vectorelement-wise moving a smoothing-window across the focus score vector,whereby at each moving step of the smoothing window, an average ormedian of the focus scores in the focus score vector elements covered bythe smoothing window are computed; generating a smoothed focus scorevector whose elements comprise the averaged or median focus scoresinstead of the original focus scores of the focus vector; identifyingthe one of the z-layers that corresponds to the element in the smoothedfocus score vector comprising the maximum focus score value as thez-layer having highest focus score.

7. The image analysis method of embodiment 5 or 6, the identification ofthe z-layer having best color separation comprising: storing the colorseparation score of each of the z-layers in a respective element of acolor separation score vector; vector element-wise moving asmoothing-window across the color separation vector, whereby at eachmoving step of the smoothing window, an average or median of the colorseparation scores in the color separation score vector elements coveredby the smoothing window are computed; generating a smoothed colorseparation score vector whose elements comprise the averaged or mediancolor separation scores instead of the original color separation scoresof the color separation vector; identifying the one of the z-layers thatcorresponds to the element in the smoothed color separation score vectorcomprising the maximum color separation score value as the z-layerhaving highest color separation score.

8. The image analysis method of anyone of the previous embodiments, thetissue sample having been stained at least with a first stain having afirst color and a second stain having a second color, the first andsecond colors differing from each other, the color separation metricbeing an indicator of color separation quality of the first and thesecond color.

9. The image analysis method of anyone of the previous embodiments, eachof the digital images being an RGB image, the computation of the colorseparation metrics comprising, for each of the digital images: computinga derivative image from the digital RGB image by computing a firstderivative value DER1 for each of the pixels in the digital RGB imageaccording to DER1=R+a_(CS)*G+b_(CS)*B, where −1≤a_(CS)≤1, where−1<b_(CS)<1, wherein R, G and B are the red, green and blue componentsof the pixel intensity of said pixel in the RGB image, wherein a_(CS)and b_(CS) are parameter values having been empirically determined asvalues which, when used for computing color separation metrics, providecolor separation metrics having predictive power in respect to colorseparation quality of an image; computing a color separation metricJ_(CS) according to the formula J_(CS)=Σ_(x)Σ_(y)(|DER1(x,y)−DER1(x,y−s)|²), wherein x and y are the x and y coordinates of thepixels in the derivative image and s is a number of pixel distancepositions, whereby s has been empirically determined as pixel distancepositions that correlate with high J_(CS) values in images of known,high color separation quality.

10. The image analysis method of embodiments 9, further comprisingempirically determining the parameters a_(CS) and b_(CS) by: obtaining aplurality of different candidate a-values and candidate b-values fora_(CS) and b_(CS), each candidate a-value and each candidate b-valuelying in the range [−1; +1]; obtaining a plurality of training z-layersin a z-stack of training digital images, each training digital imagebeing an RGB image having assigned a color separation score known toaccurately indicate the color separation quality of the training image;for each possible combination of a candidate a-value and a candidateb-value, and for each of the training images in the training z-stack,computing a derivative image according to the formulaDERl_(acand-bcand)=R+a_(CS-cand)*G+b_(CS-cand)*B, wherein R, G and B arethe red, green and blue components of the pixel intensity of said pixelin the RGB training image, wherein a_(CS-cand) and b_(CS-cand) are oneof the possible combinations of a candidate a-value and a candidate bvalue, and wherein DER1 _(acand-bcand) is the first derivative valuecomputed for each pixel in the derivative image by using said onecombination of candidate a- and b-values; for each of said computedderivative images, computing a color separation metricJ_(CS.acand-bacand) according to the formulaJ_(CS.acand-bcand)=Σ_(x)Σ_(y)(|DER1 _(acand-bacand)(x,y)−DER1_(acand-bacand) (x,y−s)|²), wherein x and y are the x and y coordinatesof the pixels in the derivative image computed for one of the trainingimages and for one combination of an a-candidate value and a b-candidatevalue, and s is a number of pixel distance positions, whereby s has beenempirically determined as pixel distance positions that correlate withhigh J_(CS) values in images of known, high color separation quality;identifying the combination of candidate a- and b-values that correlatebest with the color separation scores assigned to each of the trainingimages and known to accurately indicate the color separation quality ofsaid training image; and using the identified candidate a- and candidateb values as the parameter values a_(CS), b_(CS) having been empiricallydetermined.

11. The image analysis method of anyone of the previous embodiments,each of the digital images being an RGB image, the computation of thefocus metrics comprising, for each of the digital images: computing anoptical-density-domain image from the digital RGB image by computing aROD, a GOD and a BOD value for each pixel of the RGB image according to:ROD=−log(R/BK), GOD=−log(G/BK), BOD=−log(B/BK), wherein R, G and B arethe red, green and blue components of the pixel intensity of said pixelin the RGB image, wherein BK is a background intensity value typical fortissue slide glass; computing a derivative image from the optical domainimage by computing a second derivative value DER2 for each of the pixelsin the optical-density-domain image according toDER2=ROD+a_(F)*GOD+b_(F)*BOD, where −1≤a_(F)≤1, where −1≤b_(F)≤1,wherein a_(F) and b_(F) are parameters having been empiricallydetermined as values which, when used for computing focus metrics,provide focus metrics having predictive power in respect to whether animage is in-focus; computing a focus metric J_(F) according to theformula J_(F)=Σ_(x)Σ_(y) (|DER2(x,y)−DER2(x,y−s)|², wherein x and y arethe x and y coordinates of the pixels in the derivative image and s is anumber of pixel distance positions, whereby s has been empiricallydetermined as pixel distance positions that correlate with high J_(F)values in images known to be in-focus.

12. The image analysis method of embodiment 11, further comprisingempirically determining the parameters a_(F) and b_(F) by: obtaining aplurality of different candidate a-values and candidate b-values fora_(F) and b_(F), each candidate a-value and each candidate b-value lyingin the range [−1; +1]; obtaining a plurality of training z-layers in az-stack of training digital images, each training digital image being anRGB image having assigned a focus score known to accurately indicatewhether the training image is in-focus; computing a trainingoptical-density-domain image for each of the training images bycomputing a ROD, a GOD and a BOD value for each pixel of the trainingimage; for each possible combination of a candidate a-value and acandidate b-value, and for each of the training optical-density-domainimages in the training z-stack, computing a derivative image accordingto the formula DER2 _(acand-bcand)=ROD+a_(F-cand)*GOD+b_(F-cand)*BOD,wherein ROD, GOD and BOD respectively are components of a pixel in thetraining optical-density-domain image, wherein a_(F-cand) and b_(F-cand)are one of the possible combinations of a candidate a-value and acandidate b-value and wherein DER2 _(acand-bcand) is the secondderivative value computed for each pixel in the derivative image byusing said one combination of candidate a- and b-values; for each ofsaid computed derivative images, computing a focus metricJ_(F acand-bacand) according to the formula J_(F.acand-bcand)=Σ_(x)Σ_(y)(|DER2_(acand-bacand) (x,y)−DER2_(acand-bacand) (x,y−s)|²), wherein xand y are the x and y coordinates of the pixels in the derivative imagecomputed for one of the training images and for one combination of ana-candidate value and a b-candidate value, and s is a number of pixeldistance positions, whereby s has been empirically determined as pixeldistance positions that correlate with high J_(F) values in images ofknown focus quality; identifying the combination of candidate a_(F-cand)and b_(F-cand)−values that correlate best with the focus scores assignedto each of the training images and known to accurately indicate thefocus quality of said training image; and using the identified candidatea_(F-cand)—and candidate b_(F-cand) values as the parameter valuesa_(F), b_(F) having been empirically determined.

13. The image analysis method of any one of the previous embodiments,further comprising: obtaining a plurality of training z-layers in az-stack of training digital images, each training digital image being anRGB image having assigned a color separation score known to accuratelyindicate the color separation quality of the training image; for each ofthe training digital images computing a plurality of candidate colorseparation metrics, the plurality of candidate color separation metricscomprising metrics being incapable of indicating color separationquality and comprising metrics indicating color separation quality; foreach of the candidate color separation metrics, generating a plot (902)comprising an x-axis and a first and a second y-axis, the x-axisrepresenting the z-axis of the z-stack of training images, the firsty-axis representing units of the color separation scores assigned to thetraining images, the second y-axis representing units of the candidatecolor separation metrics, the plot generation comprising plotting thecolor separation quality scores assigned to the training images over thex-axis and the first y-axis and plotting the computed candidate colorseparation metrics computed for the training images over the x-axis andthe second y-axis; selectively using the candidate color separationmetrics whose change across the z-layers correlates with the change ofthe color separation scores assigned to the training images across thez-layers for computing the color separation metrics for the singletissue sample.

14. The image analysis method of any one of the previous embodiments,further comprising: obtaining a plurality of training z-layers in az-stack of training digital images, each training digital image being anRGB image having assigned a focus score known to accurately indicate thefocus quality of the training image; for each of the training digitalimages computing a plurality of candidate focus metrics, the pluralityof candidate focus metrics comprising metrics being incapable ofindicating focus quality and comprising metrics indicating focusquality; for each of the candidate focus metrics, generating a plot(902) comprising an x-axis and a first and a second y-axis, the x-axisrepresenting the z-axis of the z-stack of training images, the firsty-axis representing units of the focus scores assigned to the trainingimages, the second y-axis representing units of the candidate focusmetrics, the plot generation comprising plotting the focus qualityscores (904) assigned to the training images over the x-axis and thefirst y-axis and plotting the candidate focus metrics (906) computed forthe training images over the x-axis and the second y-axis; selectivelyusing the candidate focus metrics whose change across the z-layerscorrelates with the change of the focus scores assigned to the trainingimages across the z-layers for computing the focus metrics for thesingle tissue sample.

15. The image analysis method of anyone of the previous embodiments, thedetermining of the one of the z-layers to be used for computingtissue-related or cell-related scores being performed while a process ofscanning a slide comprising the tissues sample for generating thez-stack of images of said tissue sample is in progress.

16. The image analysis method according to any one of the previousembodiments, the method comprising: identifying a plurality of tiles inthe z-stack of digital images of the single tissue sample, each tilecomprising, for each of the z-layers, a sub region of the image; foreach of the tiles and for each of the z-layers, computing a plurality oftile focus metrics, each tile focus metric being an indicator of focusquality of the sub region of the digital image of said z-layer withinsaid tile and being derived from image data of said sub region; for eachof the tiles and for each of the z-layers, computing a plurality of tilecolor separation metrics, each tile color separation metric being anindicator of color separation quality of the sub region of the digitalimage of said z-layer within said tile and being derived from image dataof said sub region; evaluating the tile focus metrics and the tile colorseparation metrics of all the tiles and selecting a sub-set of the tilesin dependence on the tile focus metrics and tile color separationmetrics of the tiles; and selectively using the sub-set of the tiles foradditional automated image analysis operations and/or highlighting thesub-set of the tiles in a graphical user interface.

17. The image analysis method of embodiment 16, the method furthercomprising: automatically identifying a plurality of first dots and aplurality of second dots in each of the images of the z-stack of images,each dot being a blob of adjacent pixels, the pixel intensity values ofthe first dots correlating with an optical signal intensity of a firststain, the pixel intensity values of the second dots correlating with anoptical signal intensity of a second stain, the first and second stainhaving been used for staining the single tissue sample and havingdifferent colors.

18. The image analysis method of embodiment 17, the method furthercomprising computing one or more of the tile focus metrics by: applyinga plurality of pairs of Gaussian filters on each of the digital imagesof the z-layer of images, the kernels of the Gaussian filters havingdifferent standard deviations in each pair, for computing, for each ofthe z-layer images and for each of the pairs of Gaussian filters, adifference-of-Gaussian image, identifying, in each of thedifference-of-Gaussian images, a plurality of DoG dots, a DoG dot beinga set of adjacent pixels in a difference-of-Gaussian image whose pixelvalue exceeds a DoG threshold; computing, for each of the tiles and forall sub regions of z-stack images covered by said tile, anaverage-DoG-dot-value for all DoG dots identified in thedifference-of-Gaussian image derived from said image sub-region, theaverage-DoG-dot value being indicative of the average DoG value of allpixels in the difference-of-Gaussian image contained in said DoG dot;computing, for each of the tiles and for all sub regions of z-stackimages covered by said tile, the mean of all average-DoG-dot-valuescomputed for said tile and using said mean of all average-DoG-dot-valuesas one of the tile focus metrics; and/or computing, for each of thetiles and for all sub regions of z-stack images covered by said tile,the median of all average-DoG-dot-values computed for said tile andusing said median of all average-DoG-dot-values as one of the tile focusmetrics; and/or sorting the average-DoG-dot-values computed for each ofthe tiles and for each of the z-layers of said tile and computing, foreach of the tiles and for all sub regions of z-stack images covered bysaid tile, the mean of a predefined percentile value of the highestranking ones of the sorted average-DoG-dot-values computed for said tileand using said mean of all average-DoG-dot-values as one of the tilefocus metrics.

19. The image analysis method of embodiment 18, the method furthercomprising computing one or more of the tile focus metrics by:identifying, in each of the difference-of-Gaussian images, a pluralityof first DoG dots, a first DoG dot being a set of adjacent pixels in adifference-of-Gaussian image whose pixel value exceeds a DoG thresholdand which overlays one of the identified first dots; identifying, ineach of the difference-of-Gaussian images, a plurality of second DoGdots, a second DoG dot being a set of adjacent pixels in adifference-of-Gaussian image whose pixel value exceeds a DoG thresholdand which overlays one of the identified second dots; computing, foreach of the tiles and for all sub regions of z-stack images covered bysaid tile, an average-first-DoG-dot-value for all first DoG dotsidentified in the difference-of-Gaussian image derived from said imagesub-region, the average-first-DoG-dot value of a first DoG dot beingindicative of the average DoG value of all pixels in thedifference-of-Gaussian image contained in said first DoG dot; computing,for each of the tiles and for all sub regions of z-stack images coveredby said tile, an average-second-DoG-dot-value for all second DoG dotsidentified in the difference-of-Gaussian image derived from said imagesub-region, the average-second-DoG-dot value of a second DoG dot beingindicative of the average DoG value of all pixels in thedifference-of-Gaussian image contained in said second DoG dot;computing, for each of the tiles and for all sub regions of z-stackimages covered by said tile, the mean of allaverage-first-DoG-dot-values computed for said tile and using said meanof all average-first-DoG-dot-values as one of the tile focus metrics;and/or computing, for each of the tiles and for all sub regions ofz-stack images covered by said tile, the median of allaverage-second-DoG-dot-values computed for said tile and using saidmedian of all average-second-DoG-dot-values as one of the tile focusmetrics.

20. The image analysis method of any one of embodiments 17-19, themethod further comprising computing one or more of the tile colorseparation metrics by: representing each of the identified first andsecond dots as a dot-vector (920-934) in a color space (960); findingthe spherical coordinates of the dot-vector of each of the first andsecond dots for identifying, for each of the first and second dots, anazimuth angle (972) and an elevation angle (974) of the dot-vectorrepresented in spherical coordinates; for each of the tiles and for eachz-layer of said tiles, plotting the azimuth angles and elevation anglesof all identified first and second dots contained in anazimuth-elevation plot (990); fitting a first ellipse around all datapoints in the azimuth-elevation plot (990) having been derived from oneof the first dots; fitting a second ellipse around all data points inthe azimuth-elevation plot (990) having been derived from one of thesecond dots; computing the distance of the first and second ellipse; andusing the computed distance as one of the tile color separation metrics,whereby the larger the distance between the first and second ellipse,the higher the color separation quality.

21. The image analysis method of embodiment 20, wherein representingeach of the identified first and second dots as a dot-vector in a colorspace comprises: for each of the identified first and second dots,computing an R_(DOTAVG) value, a G_(DOTAVG)—value and a B_(DOTAVG)value, the R_(DOTAVG) value being the average of the R component of allpixels contained in said dot, the G_(DOTAVG) value being the average ofthe G component of all pixels contained in said dot, the B_(DOTAVG)value being the average of the B component of all pixels contained insaid dot, the RGB values being red green and blue pixel values in RGBcolor space; for each of the identified first and second dots, computingan optical density vector RGBOD (920-934) comprising the elementsROD_(DOTAVG), GOD_(DOTAVG) and BOD_(DOTAVG) computed for the dotaccording to: ROD_(DOTAVG)=−log(R_(DOTAVG)/BK),GOD_(DOTAVG)=−log(G_(DOTAVG)/BK), BOD_(DOTAVG)=−log(B_(DOTAVG)/BK),Using the optical density vectors RGBOD as the dot-vector of the firstand second dots.

22. The image analysis method of any one of embodiments 17-21, the tilecolor separation metrics further comprising one or more of: the maximumvalue (Amax) observed in all pixels of a dot whose RGB values arerepresented in a color channel, the color channel being a color channelin the LAB-color space whose color is the most similar to the color ofthe first or the second stain; the maximum value (Asigmax) observed inall pixels of a dot whose RGB values are represented in a color channelderivative, the color channel derivative being computed as a derivativeof a color channel in the LAB-color space whose color is the mostsimilar to the color of the first or the second stain, the computationof the derivative color channel comprising applying a sigmoid functionfor removing the effect of extreme values; the maximum color channelvalue observed in all pixels of a dot whose RGB values are representedin said color channel, the color channel having been generated by acolor deconvolution of the RGB values into at least the first color ofthe first stain and the second color of the second stain, the colorchannel generated by the color deconvolution corresponding to the firstor the second color; a radial symmetry value indicating radial symmetryof the dot; an average DoG value computed from the DoG values of allpixels of a dot, the DoG values of the Dot being obtained from adifference-of-Gaussian image area in a difference-of-Gaussian imagecorresponding to the location of the pixels of said dot.

23. The image analysis method of any one of embodiments 17-22, themethod further comprising generating a heat map for the tiles, the heatmap being indicative of the focus quality and/or of the color separationquality computed for each of the tiles, the highlighting of the sub-setof the tiles in a graphical user interface comprising displaying thecomputed heat map as an overlay of the tiles.

24. The image analysis method of any one of embodiments 17-23, each tilehaving a width and height of 250-350 pixels, wherein preferentially 1pixel corresponds to 0.23 μm of the tissue slide.

25. The image analysis method of any one of embodiments 17-24, wherein afocus quality scores and a color separation quality score are computedonly for those tiles having at least one first dot, each first dotcorresponding to an in situ hybridization signal having a first colorand at least one second dot, each second dot corresponding to an in situhybridization signal having a second color.

26. An image analysis system comprising one or more processors and astorage medium comprising computer-interpretable instructions which,when executed by the one or more processors, cause the one or moreprocessors to perform a method according to any one of embodiments 1-25.

27. A system comprising the image analysis system of any one ofembodiments 1-16 and comprising an optical slide scanning system, theslide-scanning system being configured for generating the z-stack of thedigital images of the single biological sample, the instructions, whenexecuted by the one or more processors, cause the one or more processorsto perform the method according to any one of embodiments 1-15 in realtime for already generated parts of the z-stack of the digital imageswhile the slide-scanning system scans the tissue slide.

28. A non-volatile storage medium comprising instructions which, whenexecuted by a processor, cause the processor to perform the method ofany one of embodiments 1-25.

29. A computer system for determining the most suitable z-layer in agiven z-stack, comprising one or more processors and at least onememory, the at least one memory storing non-transitory computer-readableinstructions for execution by the one or more processors to cause theone or more processors to: compute focus metrics and color separationmetrics for each z-layer within a z-stack of images, each z-layer withinthe z-stack of images corresponding to an image of a tissue sample, andevaluate the focus metrics and color separation metrics to determine amost suitable z-layer within the z-stack.

30. The computer system of embodiment 29, wherein the focus metricscomprise a focus quality score for each z-layer, and wherein the colorseparation metrics comprise a color separation quality score for eachz-layer.

31. The computer system of embodiment 30, wherein the focus qualityscore for each z-layer and the color separation quality score for eachz-layer are independently computed within empirically determined colorspaces optimized for signals of an in situ hybridization assay appliedto the tissue sample.

32. The computer system of embodiment 30, wherein the evaluation of thefocus metrics and color separation metrics comprises computing anabsolute value metric and determining whether the absolute value metricis greater than, equal to, or less than a predetermined threshold value.

33. The computer system of embodiment 31, wherein the absolute valuemetric is an absolute value of the difference between the z-layer havingbest focus and the z-layer having best color separation.

34. The computer system of embodiment 33, wherein the z-layer havingbest focus and the z-layer having best color separation are eachindependently computed by median filtering the focus quality scores andcolor separation quality scores, respectively, and then identifying amaximum value for the median filtered focus quality scores and a maximumvalue for the median filtered color separation quality scores.

35. The computer system of embodiment 34, wherein if the absolute metricis determined to be less than or equal to the predetermined thresholdvalue, instructions are provided to set the most suitable z-layer as acompromise layer metric, wherein the compromise layer metric is anaverage value of the z-layer having best focus and the z-layer havingbest color separation.

36. The computer system of embodiment 34, wherein if the absolute metricis determined to be greater than the predetermined threshold value,instructions are provided to evaluate whether the most suitable z-layershould be guided by focus features or color separation features.

37. The computer system of embodiment 36, wherein the evaluation ofwhether the most suitable z-layer should be guided by focus features orcolor separation features is determined by comparing a layer focuscomparator value to a layer color separation comparator value, wherebyif the layer focus comparator value is greater than the layer colorseparation value, the most suitable z-layer is set as a z-layer havingbest focus, and whereby if the layer focus comparator value is less thanthe layer color separation value, the most suitable z-layer is set as az-layer having best color separation.

38. A computer-implemented method for determining the most suitablez-layer in a given z-stack, comprising computing focus metrics and colorseparation metrics for each z-layer within a z-stack of images, eachz-layer within the z-stack of images corresponding to an image of atissue sample, and evaluating the focus metrics and color separationmetrics to determine a most suitable z-layer within the z-stack.

39. The computer-implemented method of embodiment 38, wherein the focusmetrics comprise a focus quality score for each z-layer, and wherein thecolor separation metrics comprise a color separation quality score foreach z-layer.

40. The computer-implemented method of embodiment 39, wherein the focusquality score for each z-layer and the color separation quality scorefor each z-layer are independently computed within empiricallydetermined color spaces, optimized for signals of an in situhybridization assay applied to the tissue sample.

41. The computer-implemented method of embodiment 40, wherein theevaluation of the focus metrics and color separation metrics comprisescomputing an absolute value metric and determining whether the absolutevalue metric is greater than, equal to, or less than a predeterminedthreshold value.

42. The computer-implemented method of embodiment 41, wherein theabsolute value metric is an absolute value of the difference between thez-layer having best focus and the z-layer having best color separation.

43. The computer-implemented method of embodiment 42, wherein thez-layer having best focus and the z-layer having best color separationare each independently computed by median filtering the focus qualityscores and color separation quality scores, respectively, and thenidentifying a maximum value for the median filtered focus quality scoresand a maximum value for the median filtered color separation qualityscores.

44. The computer-implemented method of embodiment 43, wherein if theabsolute metric is determined to be less than or equal to thepredetermined threshold value, instructions are provided to set the mostsuitable z-layer as a compromise layer metric, wherein the compromiselayer metric is an average value of the z-layer having best focus andthe z-layer having best color separation.

45. The computer-implemented method of embodiment 43, wherein if theabsolute metric is determined to be greater than the predeterminedthreshold value, instructions are provided to evaluate whether the mostsuitable z-layer should be guided by focus features or color separationfeatures.

46. The computer method of embodiment 45 wherein the evaluation ofwhether the most suitable z-layer should be guided by focus features orcolor separation features is determined by comparing a layer focuscomparator value to a layer color separation comparator value, wherebyif the layer focus comparator value is greater than the layer colorseparation value, the most suitable z-layer is set as a z-layer havingbest focus, and whereby if the layer focus comparator value is less thanthe layer color separation value, the most suitable z-layer is set as az-layer having best color separation.

47. A computer-implemented method for the automated evaluation of imagetiles derived from a whole slide scan comprising: (a) computing aplurality of focus features and a plurality of color separation featuresfor each individual image tile; (b) deriving a focus quality score fromthe plurality of focus features and a color separation quality scorefrom the plurality of color separation features; and (c) identifyingdigital image tiles more suitable for downstream processing based on thefocus quality score and the color separation quality score.

48. The computer-implemented method of embodiment 47, wherein theplurality of focus features are selected from (i) a mean/median of the0th DoG layer for all dot pixels in the image tile; (ii) a mean/medianof the 0th DoG layer for all black dot pixels in the image tile; (iii) amean/median of the max DoG values of the 0th layer for all dot blobs inthe image tile; and (iv) a mean of the top DoG values for the 0th layerfor all dot blobs in the image tile.

49. The computer-implemented method of embodiment 47, wherein theplurality of color separation features are selected from (i) Amax, (ii)Asigmax; (iii) a maximum in an unmixed black channel; (iv) a gradientvalue for at least one channel selected from the group consisting of aluminance channel, an unmixed red channel, an A channel, an Asigchannel, and a green channel; (v) DoG values in at least one of achannel selected from the group consisting of a green channel, an Achannel, and an optical density domain absorbance channel; and (vi)color domain features based on modeling color information in terms ofellipses.

50. The computer-implemented method of embodiment 47, further comprisinggenerating a heat map.

51. The computer-implemented method of embodiment 47, further comprisinggenerating an overlay, where the overlay indicates digital image tilesmore suitable for downstream processing.

52. The computer-implemented method of embodiment 47, wherein the focusquality scores and color separation quality score are computed only forthose tiles having at least one dot corresponding to a first in situhybridization signal and at least one dot corresponding to a second insitu hybridization signal.

53. A computer system for the automated evaluation of image tilesderived from a whole slide scan comprising one or more processors and atleast one memory, the at least one memory storing non-transitorycomputer-readable instructions for execution by the one or moreprocessors to cause the one or more processors to: (a) compute aplurality of focus features and a plurality of color separation featuresfor each individual image tile; (b) derive a focus quality score fromthe plurality of focus features and a color separation quality scorefrom the plurality of color separation features; and (c) identifydigital image tiles more suitable for downstream processing based on thefocus quality score and the color separation quality score.

54. The computer system of embodiment 53, wherein the plurality of focusfeatures are selected from (i) a mean/median of the 0th DoG layer forall dot pixels in the image tile; (ii) a mean/median of the 0th DoGlayer for all black dot pixels in the image tile; (iii) a mean/median ofthe max DoG values of the 0th layer for all dot blobs in the image tile;and (iv) a mean of the top DoG values for the 0th layer for all dotblobs in the image tile.

55. The computer system of embodiment 53, wherein the plurality of colorseparation features are selected from (i) Amax, (ii) Asigmax; (iii) amaximum in an unmixed black channel; (iv) a gradient value for at leastone channel selected from the group consisting of a luminance channel,an unmixed red channel, an A channel, an Asig channel, and a greenchannel; (v) DoG values in at least one of a channel selected from thegroup consisting of a green channel, an A channel, and an opticaldensity domain absorbance channel; and (vi) color domain features basedon modeling color information in terms of ellipses.

56. The computer system of embodiment 53, wherein instructions areprovided to compute a heat map.

57. The computer system of embodiment 53, wherein instructions areprovided to generate an overlay, where the overlay indicates digitalimage tiles more suitable for downstream processing.

58. The computer system of embodiment 53, wherein the focus qualityscores and color separation quality score are computed only for thosetiles having at least one dot corresponding to a first in situhybridization signal and at least one dot corresponding to a second insitu hybridization signal.

59. A computer device configured to analyze images captured from tissuesamples comprising one or more processors and at least one memory, theat least one memory storing non-transitory computer-readableinstructions for execution by the one or more processors to cause theone or more processors to (i) receive a series of images as input, eachimage corresponding to a z-layer in a z-stack; (ii) run a focusassessment module to compute a number of metrics derived from focusfeatures and color separation features within the images; and (iii)output a most suitable z-layer in a z-stack of images based on thecomputed metrics.

60. The computer device of embodiment 53, wherein a quality assessmentmodule is run and an identification of tiles within the input image thatare more suitable for downstream processing are provided.

61. A computer system for determining a z-layer that, compared to otherz-layers in a given z-stack, comprise features that are well focused andhave clearly discernible color features such that differently coloredfeatures may be recognized from each other, comprising one or moreprocessors and at least one memory, the at least one memory storingnon-transitory computer-readable instructions for execution by the oneor more processors to cause the one or more processors to executeinstructions to: retrieve a series of images from a z-stack of images,each image corresponding to a different z-layer in the z-stack, andwherein the images are captured from a portion of a tissue specimenpre-treated with two in situ hybridization probes; compute focus metricsand color separation metrics for each z-layer in the z-stack, whereinthe focus metrics are derived from focus features of stained regionswithin the images and wherein the color separation metrics are derivedfrom color separation features of stained regions within the images; anddetermine the z-layer that, compared to other z-layers in the givenz-stack comprises features that are well focused and have clearlydiscernible color features, based on evaluation of the focus metrics andcolor separation metrics.

All of the U.S. patents, U.S. patent application publications, U.S.patent applications, foreign patents, foreign patent applications andnon-patent publications referred to in this specification and/or listedin the Application Data Sheet are incorporated herein by reference, intheir entirety. Aspects of the embodiments can be modified, if necessaryto employ concepts of the various patents, applications and publicationsto provide yet further embodiments.

Although the disclosure herein has been described with reference toparticular embodiments, it is to be understood that these embodimentsare merely illustrative of the principles and applications of thepresent disclosure. It is therefore understood that numerousmodifications may be made to the illustrative embodiments and that otherarrangements may be devised without departing from the spirit and scopeof the present disclosure as defined by the appended claims.

The invention claimed is:
 1. An image analysis method for determiningone of a plurality of z-layers in a z-stack of digital images to be usedfor computing tissue-related or cell-related scores, the z-stack ofdigital images depicting different layers of a single tissue samplelocated on a slide, the tissue sample having been stained at least witha first stain having a first color and a second stain having a secondcolor, the first and second colors differing from each other, thez-stack images having been acquired by varying the focus of an opticalsensor, the method being implemented in an image analysis system andcomprising: for each of the z-layers, computing focus metrics, eachfocus metric being an indicator of focus quality of the z-layer andbeing derived from image data of said z-layer; for each of the z-layers,computing color separation metrics, each color separation metric beingan indicator of color separation quality of the z-layer and beingderived from image data of said z-layer; and evaluating the focusmetrics and the color separation metrics and determining the one of thez-layers to be used for computing tissue-related or cell-related scoresin dependence on the focus metrics and color separation metrics of eachof the z-layers.
 2. The image analysis method of claim 1, furthercomprising: analyzing selectively the one of the images in the z-stackof digital images that corresponds to the determined z-layer andselectively computing the tissue-related or cell-related scores for saidanalyzed image.
 3. The image analysis method of claim 1, wherein theevaluation of the focus metrics and the color separation metricscomprises: computing the difference between the z-layer having bestfocus and the z-layer having best color separation by computing anabsolute value of the difference of the index position of the z-layerhaving best focus and the index position of the z-layer having bestcolor separation in the z-stack; determining whether the computeddifference is greater than, equal to, or less than a predeterminedthreshold value; in case the computed difference is determined to beless than or equal to the predetermined threshold value: computing anaverage of the z-layer index of the z-layer having best focus and thez-layer index of the z-layer having best color separation; if theaverage of the z-layer index is an integer, identifying the one of thez-layers of the z-stack whose index position is identical to thecomputed average of the z-layer indices; if the average of the z-layerindex is not an integer, identifying the two consecutive z-layers of thez-stack which lie next to the computed average of the z-layer indicesand identifying the one of the two consecutive z-layers having thehighest color separation metrics; and using said one identified z-layeras the one z-layer to be used for computing the scores; and in case thecomputed difference is determined to be greater than the predeterminedthreshold value: comparing a focus score derived from the focus metricsof the z-layer image having best focus with a reference focus score, thereference focus score being an empirically determined focus scoreobtained for a digital image depicting the same type of tissue sampleand having been manually identified as being in-focus for determining adeviation of the focus score from the reference focus score; comparing acolor separation score derived from the color separation metrics of thez-layer image having best color separation with a reference colorseparation score, the reference color separation score being anempirically determined color separation score obtained for a digitalimage depicting the same type of tissue sample and having been manuallyidentified as being a high quality color separation image fordetermining a deviation of the color separation score from the referencecolor separation score; in case the deviation of the focus score fromthe reference focus score is greater than the deviation of the colorseparation score from the reference color separation score, using thez-layer image having best color separation for computing the scores. 4.The image analysis method of claim 3, further comprising: in case thedeviation of the color separation score and the reference colorseparation score is greater than the deviation of the focus score andthe reference focus score, using the z-layer image having best focus forcomputing the scores.
 5. The image analysis method of claim 1, furthercomprising: computing, for each of the z-layers, a focus score by usingthe focus metrics of the digital image corresponding to the z-layer asinput; computing, for each of the z-layers, a color separation score byusing the color separation metrics of the digital image corresponding tothe z-layer as input; identifying the z-layer having best focus andidentifying the z-layer having best color separation.
 6. The imageanalysis method of claim 5, the identification of the z-layer havingbest focus comprising: storing the focus score of each of the z-layersin a respective element of a focus score vector; vector element-wisemoving a smoothing-window across the focus score vector, whereby at eachmoving step of the smoothing window, an average or median of the focusscores in the focus score vector elements covered by the smoothingwindow are computed; generating a smoothed focus score vector whoseelements comprise the averaged or median focus scores instead of theoriginal focus scores of the focus vector; identifying the one of thez-layers that corresponds to the element in the smoothed focus scorevector comprising the maximum focus score value as the z-layer havinghighest focus score.
 7. The image analysis method of claim 5, theidentification of the z-layer having best color separation comprising:storing the color separation score of each of the z-layers in arespective element of a color separation score vector; vectorelement-wise moving a smoothing-window across the color separationvector, whereby at each moving step of the smoothing window, an averageor median of the color separation scores in the color separation scorevector elements covered by the smoothing window are computed; generatinga smoothed color separation score vector whose elements comprise theaveraged or median color separation scores instead of the original colorseparation scores of the color separation vector; identifying the one ofthe z-layers that corresponds to the element in the smoothed colorseparation score vector comprising the maximum color separation scorevalue as the z-layer having highest color separation score.
 8. The imageanalysis method of claim 1, each of the digital images being an RGBimage, the computation of the color separation metrics comprising, foreach of the digital images: computing a derivative image from thedigital RGB image by computing a first derivative value DER1 for each ofthe pixels in the digital RGB image according toDER1=R+a_(CS)*G+b_(CS)*B, where −1≤a_(CS)≤1, where −1≤b_(CS)≤1, whereinR, G and B are the red, green and blue components of the pixel intensityof said pixel in the RGB image, wherein a_(CS) and b_(CS) are parametervalues having been empirically determined as values which, when used forcomputing color separation metrics, provide color separation metricshaving predictive power in respect to color separation quality of animage; computing a color separation metric J_(CS) according to theformulaJ _(CS)=Σ_(x)Σ_(y)(|DER1(x,y)−DER1(x,y−s)|²), wherein x and y are the xand y coordinates of the pixels in the derivative image and s is anumber of pixel distance positions, whereby s has been empiricallydetermined as pixel distance positions that correlate with high J_(CS)values in images of known, high color separation quality.
 9. The imageanalysis method of claim 1, each of the digital images being an RGBimage, the computation of the focus metrics comprising, for each of thedigital images: computing an optical-density-domain image from thedigital RGB image by computing a ROD, a GOD and a BOD value for eachpixel of the RGB image according to:ROD=−log(R/BK), GOD=−log(G/BK), BOD=−log(B/BK), wherein R, G and B arethe red, green and blue components of the pixel intensity of said pixelin the RGB image, wherein BK is a background intensity value typical fortissue slide glass; computing a derivative image from the optical domainimage by computing a second derivative value DER2 for each of the pixelsin the optical-density-domain image according toDER2=ROD+a_(F)*GOD+b_(F)*BOD, where −1≤a_(F)≤1, where −1≤b_(F)≤1,wherein a_(F) and b_(F) are parameters having been empiricallydetermined as values which, when used for computing focus metrics,provide focus metrics having predictive power in respect to whether animage is in-focus; computing a focus metric J_(F) according to theformulaJ _(F)=Σ_(x)Σ_(y)(|DER2(x,y)−DER2(x,y−s)|²), wherein x and y are the xand y coordinates of the pixels in the derivative image and s is anumber of pixel distance positions, whereby s has been empiricallydetermined as pixel distance positions that correlate with high J_(F)values in images known to be in-focus.
 10. The image analysis method ofclaim 1, the method comprising: identifying a plurality of tiles in thez-stack of digital images of the single tissue sample, each tilecomprising, for each of the z-layers, a sub region of the image; foreach of the tiles and for each of the z-layers, computing a plurality oftile focus metrics, each tile focus metric being an indicator of focusquality of the sub region of the digital image of said z-layer withinsaid tile and being derived from image data of said sub region; for eachof the tiles and for each of the z-layers, computing a plurality of tilecolor separation metrics, each tile color separation metric being anindicator of color separation quality of the sub region of the digitalimage of said z-layer within said tile and being derived from image dataof said sub region; evaluating the tile focus metrics and the tile colorseparation metrics of all the tiles and selecting a sub-set of the tilesin dependence on the tile focus metrics and tile color separationmetrics of the tiles; and selectively using the sub-set of the tiles foradditional automated image analysis operations and/or highlighting thesub-set of the tiles in a graphical user interface.
 11. The imageanalysis method of claim 10, the method further comprising:automatically identifying a plurality of first dots and a plurality ofsecond dots in each of the images of the z-stack of images, each dotbeing a blob of adjacent pixels, the pixel intensity values of the firstdots correlating with an optical signal intensity of a first stain, thepixel intensity values of the second dots correlating with an opticalsignal intensity of a second stain, the first and second stain havingbeen used for staining the single tissue sample and having differentcolors.
 12. The image analysis method of claim 11, the method furthercomprising computing one or more of the tile focus metrics by: applyinga plurality of pairs of Gaussian filters on each of the digital imagesof the z-layer of images, the kernels of the Gaussian filters havingdifferent standard deviations in each pair, for computing, for each ofthe z-layer images and for each of the pairs of Gaussian filters, adifference-of-Gaussian image, identifying, in each of thedifference-of-Gaussian images, a plurality of DoG dots, a DoG dot beinga set of adjacent pixels in a difference-of-Gaussian image whose pixelvalue exceeds a DoG threshold; computing, for each of the tiles and forall sub regions of z-stack images covered by said tile, anaverage-DoG-dot-value for all DoG dots identified in thedifference-of-Gaussian image derived from said image sub-region, theaverage-DoG-dot value being indicative of the average DoG value of allpixels in the difference-of-Gaussian image contained in said DoG dot;computing, for each of the tiles and for all sub regions of z-stackimages covered by said tile, the mean of all average-DoG-dot-valuescomputed for said tile and using said mean of all average-DoG-dot-valuesas one of the tile focus metrics; and/or computing, for each of thetiles and for all sub regions of z-stack images covered by said tile,the median of all average-DoG-dot-values computed for said tile andusing said median of all average-DoG-dot-values as one of the tile focusmetrics; and/or sorting the average-DoG-dot-values computed for each ofthe tiles and for each of the z-layers of said tile and computing, foreach of the tiles and for all sub regions of z-stack images covered bysaid tile, the mean of a predefined percentile value of the highestranking ones of the sorted average-DoG-dot-values computed for said tileand using said mean of all average-DoG-dot-values as one of the tilefocus metrics.
 13. The image analysis method of claim 11, the methodfurther comprising computing one or more of the tile focus metrics by:identifying, in each of the difference-of-Gaussian images, a pluralityof first DoG dots, a first DoG dot being a set of adjacent pixels in adifference-of-Gaussian image whose pixel value exceeds a DoG thresholdand which overlays one of the identified first dots; identifying, ineach of the difference-of-Gaussian images, a plurality of second DoGdots, a second DoG dot being a set of adjacent pixels in adifference-of-Gaussian image whose pixel value exceeds a DoG thresholdand which overlays one of the identified second dots; computing, foreach of the tiles and for all sub regions of z-stack images covered bysaid tile, an average-first-DoG-dot-value for all first DoG dotsidentified in the difference-of-Gaussian image derived from said imagesub-region, the average-first-DoG-dot value of a first DoG dot beingindicative of the average DoG value of all pixels in thedifference-of-Gaussian image contained in said first DoG dot; computing,for each of the tiles and for all sub regions of z-stack images coveredby said tile, an average-second-DoG-dot-value for all second DoG dotsidentified in the difference-of-Gaussian image derived from said imagesub-region, the average-second-DoG-dot value of a second DoG dot beingindicative of the average DoG value of all pixels in thedifference-of-Gaussian image contained in said second DoG dot;computing, for each of the tiles and for all sub regions of z-stackimages covered by said tile, the mean of allaverage-first-DoG-dot-values computed for said tile and using said meanof all average-first-DoG-dot-values as one of the tile focus metrics;and/or computing, for each of the tiles and for all sub regions ofz-stack images covered by said tile, the median of allaverage-second-DoG-dot-values computed for said tile and using saidmedian of all average-second-DoG-dot-values as one of the tile focusmetrics.
 14. The image analysis method of claim 11, the method furthercomprising computing one or more of the tile color separation metricsby: representing each of the identified first and second dots as adot-vector in a color space; finding the spherical coordinates of thedot-vector of each of the first and second dots for identifying, foreach of the first and second dots, an azimuth angle and an elevationangle of the dot-vector represented in spherical coordinates; for eachof the tiles and for each z-layer of said tiles, plotting the azimuthangles and elevation angles of all identified first and second dotscontained in an azimuth-elevation plot ; fitting a first ellipse aroundall data points in the azimuth-elevation plot having been derived fromone of the first dots; fitting a second ellipse around all data pointsin the azimuth-elevation plot having been derived from one of the seconddots; computing the distance of the first and second ellipse; and usingthe computed distance as one of the tile color separation metrics,whereby the larger the distance between the first and second ellipse,the higher the color separation quality.
 15. The image analysis methodof claim 14, wherein representing each of the identified first andsecond dots as a dot-vector in a color space comprises: for each of theidentified first and second dots, computing an R_(DOTAVG) value, aG_(DOTAVG)—value and a B_(DOTAVG) value, the R_(DOTAVG) value being theaverage of the R component of all pixels contained in said dot, theG_(DOTAVG) value being the average of the G component of all pixelscontained in said dot, the B_(DOTAVG) value being the average of the Bcomponent of all pixels contained in said dot, the RGB values being redgreen and blue pixel values in RGB color space; for each of theidentified first and second dots, computing an optical density vectorRGBOD (920-934) comprising the elements ROD_(DOTAVG), GOD_(DOTAVG) andBOD_(DOTAVG) computed for the dot according to:ROD_(DOTAVG)=−log(R_(DOTAVG)/BK), GOD_(DOTAVG)=−log(G_(DOTAVG)/BK),BOD_(DOTAVG)=−log(B_(DOTAVG)/BK), using the optical density vectorsRGBOD as the dot-vector of the first and second dots.
 16. The imageanalysis method of claim 11, the tile color separation metrics furthercomprising one or more of: the maximum value (Amax) observed in allpixels of a dot whose RGB values are represented in a color channel, thecolor channel being a color channel in the LAB-color space whose coloris the most similar to the color of the first or the second stain; themaximum value (Asigmax) observed in all pixels of a dot whose RGB valuesare represented in a color channel derivative, the color channelderivative being computed as a derivative of a color channel in theLAB-color space whose color is the most similar to the color of thefirst or the second stain, the computation of the derivative colorchannel comprising applying a sigmoid function for removing the effectof extreme values; the maximum color channel value observed in allpixels of a dot whose RGB values are represented in said color channel,the color channel having been generated by a color deconvolution of theRGB values into at least the first color of the first stain and thesecond color of the second stain, the color channel generated by thecolor deconvolution corresponding to the first or the second color; aradial symmetry value indicating radial symmetry of the dot; an averageDoG value computed from the DoG values of all pixels of a dot, the DoGvalues of the Dot being obtained from a difference-of-Gaussian imagearea in a difference-of-Gaussian image corresponding to the location ofthe pixels of said dot.
 17. The image analysis method of claim 11, themethod further comprising generating a heat map for the tiles, the heatmap being indicative of the focus quality and/or of the color separationquality computed for each of the tiles, the highlighting of the sub-setof the tiles in a graphical user interface comprising displaying thecomputed heat map as an overlay of the tiles.
 18. The image analysismethod of any one of claims 11, wherein a focus quality scores and acolor separation quality score are computed only for those tiles havingat least one first dot, each first dot corresponding to an in situhybridization signal having a first color and at least one second dot,each second dot corresponding to an in situ hybridization signal havinga second color.
 19. A non-transitory computer-readable medium storingcomputer-executable instructions, which, when executed by one or moreprocessors, cause the one or more processors to: compute focus metricsand color separation metrics for each z-layer within a z-stack ofimages, each z-layer within the z-stack of images corresponding to animage of a tissue sample, the focus metrics comprising a focus qualityscore for each z-layer, and the the color separation metrics comprisinga color separation quality score for each z-layer, and wherein the focusquality score for each z-layer and the color separation quality scorefor each z-layer are independently computed within empiricallydetermined color spaces optimized for signals of an in situhybridization assay applied to the tissue sample, and evaluate the focusmetrics and color separation metrics to determine a most suitablez-layer within the z-stack.
 20. The non-transitory computer-readablemedium of claim 19, wherein the evaluation of the focus metrics andcolor separation metrics comprises computing an absolute value metricand determining whether the absolute value metric is greater than, equalto, or less than a predetermined threshold value, and wherein theabsolute value metric is an absolute value of the difference between thez-layer having best focus and the z-layer having best color separation.21. The non-transitory computer-readable medium of claim 20, wherein thez-layer having best focus and the z-layer having best color separationare each independently computed by median filtering the focus qualityscores and color separation quality scores, respectively, and thenidentifying a maximum value for the median filtered focus quality scoresand a maximum value for the median filtered color separation qualityscores.
 22. The non-transitory computer-readable medium of claim 21,wherein if the absolute metric is determined to be less than or equal tothe predetermined threshold value, instructions are provided to set themost suitable z-layer as a compromise layer metric, wherein thecompromise layer metric is an average value of the z-layer having bestfocus and the z-layer having best color separation.
 23. Thenon-transitory computer-readable medium of claim 22, wherein if theabsolute metric is determined to be greater than the predeterminedthreshold value, instructions are provided to evaluate whether the mostsuitable z-layer should be guided by focus features or color separationfeatures, and wherein the evaluation of whether the most suitablez-layer should be guided by focus features or color separation featuresis determined by comparing a layer focus comparator value to a layercolor separation comparator value, whereby if the layer focus comparatorvalue is greater than the layer color separation value, the mostsuitable z-layer is set as a z-layer having best focus, and whereby ifthe layer focus comparator value is less than the layer color separationvalue, the most suitable z-layer is set as a z-layer having best colorseparation.
 24. A system for the automated evaluation of image tilesderived from a whole slide scan, the imaging system comprising: (i) oneor more processors, and (ii) a memory coupled to the processor, thememory to store computer-executable instructions that, when executed bythe one or more processors, cause the one or more processors to performoperations comprising: (a) computing a plurality of focus features and aplurality of color separation features for each individual image tile;(b) deriving a focus quality score for each individual tile from theplurality of focus features and deriving a color separation qualityscore for each individual tile from the plurality of color separationfeatures, wherein the focus quality scores and color separation qualityscore are computed only for those tiles having at least one dotcorresponding to a first in situ hybridization signal and at least onedot corresponding to a second in situ hybridization signal; and (c)identifying digital image tiles more suitable for downstream processingbased on the focus quality score and the color separation quality score.25. The system of claim 24, wherein the plurality of focus features areselected from (i) a mean/median of the 0^(th) DoG layer for all dotpixels in the image tile; (ii) a mean/median of the 0^(th) DoG layer forall black dot pixels in the image tile; (iii) a mean/median of the maxDoG values of the 0^(th) layer for all dot blobs in the image tile; and(iv) a mean of the top DoG values for the 0^(th) layer for all dot blobsin the image tile.
 26. The system of claim 24, wherein the plurality ofcolor separation features are selected from (i) Amax, (ii) Asigmax;(iii) a maximum in an unmixed black channel; (iv) a gradient value forat least one channel selected from the group consisting of a luminancechannel, an unmixed red channel, an A channel, an Asig channel, and agreen channel; (v) DoG values in at least one of a channel selected fromthe group consisting of a green channel, an A channel, and an opticaldensity domain absorbance channel; and (vi) color domain features basedon modeling color information in terms of ellipses.
 27. The system ofclaim 24, further comprising generating a heat map.
 28. The system ofclaim 24, further comprising generating an overlay, where the overlayindicates digital image tiles more suitable for downstream processing.